The Model Is a Witness
Two words in a subject line decide whether a training business made money. A DocuSign email that starts with "Completed:" means someone signed a voucher and the money is real. One that starts with "Complete with Docusign:" means someone was only asked to sign. I fed both to a language model and asked whether the voucher had come in. It said yes to both. To a model, a request to sign and a signed contract are the same sentence with slightly different punctuation. That near miss gave me the rule every money-touching pipeline I run is now built on: the model can recognize the event, but it does not get to define it.
01Two words run a business
I did not find that difference in a spec or a data dictionary. I found it by reading the inbox where the business actually runs, thread by thread, the way you would read a ledger. The list of categories behind the inbox agent is hand-written from real messages: which senders carry money, which subjects are contracts, which attachments are just noise. And the one marker that matters, the one that tells booked revenue apart from a polite request, is a plain string check on the subject prefix. No temperature. No embedding. Just startsWith.
What broke
I almost booked revenue that did not exist. The model read "Complete with Docusign:" and reported the voucher as received. A plain prefix check now sits between that answer and the ledger, and it has never once gotten creative.
The model still has a seat. Just not that one. When a coach writes something human, like "signed and attached, sorry for the delay," no prefix rule will catch it. That is real ambiguity, and that is exactly the work the classifier should do.
The deterministic layer makes the law. The model gives testimony.
02Enrich first, then reason
Daniel is my denial-management agent for the medical-billing operation. Completely different industry: healthcare revenue cycle instead of training, CARC codes instead of DocuSign subjects. Same shape. Before the model reads a single denial, a deterministic layer does the defining. The CARC code is looked up against a reference database. Payer rules attach. Bundling edits apply. The timely-filing math runs to the day. Only then does the model reason, inside a frame it cannot redraw.
The evals showed how much the seating matters. When Daniel's recommendations floated free, mapped-action accuracy on the 30-case golden set was 36.7%. Tying each recommendation to the reference database's per-code action (write off, bill the patient, correct and resubmit) moved it to 76.7% in one change, and classification went from 80% to 93.3%. The failures that were left were about definitions, not perception. The model wanted to appeal a bundling denial that should be corrected and resubmitted, and it kept writing off deductible balances that the reference layer knows must go to the patient. Every fix was the same fix. Take a definition away from the model and give it to the data.
The result
With the deterministic layer making the calls, Daniel cleared the ship gate at 92.0% classification and 90.7% action accuracy against golden across 150 real denial cases. Same model as before. Different seat.
A wrong yes books money that is not there.
03Three industries, one seating chart
The rule was learned in training and proven in healthcare, and by the time I built MOTOR, my quoting engine for a water-feature contracting business, it was policy from day one. MOTOR prices jobs with the manufacturer's official formulas, behind three deterministic layers that keep it from ever quoting too low. The model never writes a number on a quote. It reads the messy site notes, the "sloped yard, maybe 16 by 11, client wants a stream" kind of thing that no formula can parse, and turns them into clean inputs. Then arithmetic does the pricing, because arithmetic does not round in the customer's favor just to be agreeable.
None of this is being down on models. Deterministic code cannot read "signed and attached," and a model cannot be trusted with "Completed:". Each one is unbeatable in its own seat and dangerous in the other's. Three industries, no shared regulators, no shared vocabulary, one shared risk: a wrong yes makes money that is not there, or a price that does not cover the job. So the categories get hand-written from real cases, the markers that matter get decided by code, and the model gets the genuinely ambiguous middle. A real job. A hard one. The only one it should have.
Key insight
In any pipeline where a mistake costs money, hand-write the categories from real cases, decide the markers that matter with plain code, and give the model only the genuinely ambiguous middle.
04What three ledgers taught me
- 1Read the real inbox before you write a prompt. The categories live in the mail, not in the model.
- 2If a string decides money, check the string. A prefix comparison costs zero tokens and never hallucinates.
- 3Enrich first, then reason. Codes first. Rules first. Deadline math first. Judgment last.
- 4Give the model only the fuzzy middle. "Signed and attached" is classifier work. "Completed:" is not.
- 5When an eval fails, ask which seat the failure was in. Most of Daniel's wrong answers were definitions the model should never have owned.
Training, healthcare billing, and outdoor construction share no regulators, no customers, and no words. They share one operator and one seating chart. And the chart holds because it was learned as a scar in one industry and carried as law into the next two. If you are building anything where a wrong yes makes money out of thin air, decide your prefixes in code, and let the model testify.