What a Right Answer Costs

My AI agent could call an insurance company for about $0.98. An offshore biller cost about $5 for the same call. I had the spreadsheet open and thought I was done. Then I added one more column: how many calls came back with an answer I would actually trust. That column made the agent the more expensive option.

01What you divide by

Every AI cost model I've seen divides by calls. Cost per call, cost per minute, cost per token. Those numbers are easy to get because the meter gives them to you.

But nobody in medical billing wants a call. They want to know if a claim was received, why it was denied, and what it will take to get it paid. The call is only how you find out. So what does finding out cost?

In July my agent got 5 to 10 right answers per 100 calls. At $0.98 a call, each right answer cost somewhere between $10 and $20. Say it got 7 right out of 100. That's $14 an answer. Now take the offshore biller at $5 a call, and assume they only get the answer half the time. That still works out to $10 an answer. The person was cheaper. I just couldn't see it, because I was dividing by the wrong thing.

02Wrong is worse than nothing

July was bad. October 21 was worse.

That day the agent didn't just fail to get answers. It got wrong ones and saved them. On 8 of its last 11 calls, it wrote results into the record that weren't true.

Why is that worse than a dead call? A dead call is honest. It shows up as a failure, someone sees it, and someone calls again. A wrong answer in the record looks like success. The claim moves forward on a false fact. Maybe a biller acts on it. More likely nobody acts at all, because the system says it's handled. By the time the money fails to show up, weeks have gone by and nobody remembers why.

So a wrong answer doesn't cost zero. If you're honest about the math, it costs less than zero.

A wrong answer in the record costs more than no answer at all.

03It isn't only billing

I build systems in a few industries that have nothing to do with each other. This same problem shows up in every one.

I build a quoting engine for water-feature contractors. A fast quote that comes out wrong is a disaster. If you under-quote a pond, you pay the difference out of your own pocket on the job. So the engine has a check that refuses to under-quote. Nobody asks how fast it quotes. They ask if they can sign the number.

The denial agent works the same way. Drafting a hundred appeals for pennies means nothing if the few that matter cite the wrong policy. What you're paying for is the appeal that gets the claim paid.

Training local models taught me the same thing again. A run that finishes fast and scores well is worth nothing if the grader is wrong. I've been fooled by a bad grader more than once.

In each case the meter counts raw output, and raw output is cheap. What matters is the one output you'd stake money on. Those two numbers can be ten times apart. You can go a long time without noticing, because the first one is on the invoice and you have to dig for the second.

04What I do now

1. 1Divide by answers you'd bet a claim on. If you can't count those, you don't know your cost yet.
2. 2Count wrong answers as negative. A saved false result has to cost more than a blank, or your system will learn that confident guessing pays.
3. 3Make it easy for the agent to say "I don't know." An honest blank is cheaper than a confident mistake.
4. 4Put the human through the same math. The offshore biller doesn't cost $5. They cost $5 divided by how often they're right.

The agent is still cheaper per call than any person I could hire. But that was never what I needed to know. I needed to know what a right answer costs. For a while, it cost more than I was paying before.