Most of It Says No
I went looking for the model call in our denial agent last month and it took me four minutes to find it. One function, about forty lines. Everything around it, eleven files, exists to decide whether that function is allowed to run at all. I had built what I thought was an AI system, and most of it was a system for not using AI.
01The same shape in three places
I run software in industries that have nothing to do with each other. One system reads insurance denials and drafts appeals. One prices water features for contractors, the pump, the liner, the rock, the labor. One calls insurers and waits on hold for forty minutes so a person does not have to. Different buyers, different rules, different words for everything.
They came out the same shape. Each one has a small part that produces an answer and a large part that decides whether the answer is allowed to leave. I did not design that. I noticed it the third time.
It goes further down than I expected. On my own hardware I run open models, Qwen and DeepSeek, and their main job is not to answer anything. They read the input and decide whether the expensive model ever sees it. A gate in front of a gate.
02What guessing costs
Early on I let the model do arithmetic in the quoting engine. It read the job, picked the parts, added up the bill of materials, and produced a number that looked completely reasonable. It was low by about nine percent. In construction, low is the direction that takes money out of the contractor's pocket, and nobody notices until the hole is already dug.
What broke
A quote went out under cost because I let a language model add. The number looked right, which is exactly why nobody checked it.
Pricing moved into plain code that afternoon. The model picks parts and writes the description. Multiplication belongs to a function that cannot be talked into anything.
The denial agent taught me the same lesson in a different accent. It once wrote a careful, well argued appeal that cited a document we did not have. Nothing was wrong with the writing. The rule now is boring: no source in hand, no sentence on the page. I did a Master's in Language and Theology before I wrote a line of production code, and the useful thing that training gives you is a nose for the gap between what a text says and what you wish it said. That turns out to be most of the job.
03Abstention is a dial
Everyone tunes accuracy. Almost nobody publishes the other number, which is how often the system declines to answer. The two move together, and the second one is the one you actually control. Our denial classifier runs at ninety percent on the cases it keeps, and it keeps about seven out of eight. I can push coverage to everything and watch accuracy fall, or refuse more and look useless. The work is choosing the point and being able to defend it.
The dial only matters if the refusal has somewhere to go. This is the part I got wrong for a year. A system that stops is not automatically a safe system.
If the machine stops and nobody gets the work, you have not built caution. You have built a hole.
So every refusal now has an address. The denial agent hands the case to a biller with the reason attached. The voice agent drops the call into a human queue with the recording and the time in Central. The quoting engine flags the line item and will not render a PDF until someone signs off on it.
The result
Since pricing left the model, no quote has gone out under cost, and the hard jobs land in a review queue instead of a customer's inbox.
Key insight
The part of an AI system that decides not to answer is what makes the rest of it safe to put in front of anyone.
04What to build first
- 1Write the refusal path before the happy path. It is harder, and you will not go back and do it later.
- 2Give every refusal an address: a queue, a person, a fallback. Silence is not a destination.
- 3Take arithmetic away from the model. Anything with a right answer belongs in code.
- 4Report coverage next to accuracy. One number without the other is marketing.
- 5Watch the gate, not the model. The gate is the part that fails quietly.
None of this is clever. It is the least interesting code I write, and it is the reason the interesting code is allowed near a customer at all. The people shipping demos are tuning the model. The people running things are mostly building the part that says no, and they are mostly doing it alone, which is a shame, because the shape is the same in every industry I have worked in. If you are building that part, I would like to compare notes.