Nobody Asked the Model
For months my eval dashboard said my skills scored 0 percent, and I believed it. Zero is a believable number for a young system. So I rewrote prompts. I restructured skills. I tightened trigger phrases, reran the suite, and watched the zero not move. The whole time, no model was ever asked a single question. The harness had quietly swapped credentials on me. Every request died at the front door with a 401, and every 401 was scored as a wrong answer. I spent months improving the answers of a model nobody asked.
01One line of fallback
The bug was one line. Line 90 of run_eval.py had a fallback: if no API key was set, use the OAuth token from my Claude Code session as an API key. It looks harmless. It is the kind of line you write at midnight so a script will run on a fresh machine. But an OAuth token is not an API key. The API rejected every request with a 401, the harness caught the failure, and it had nowhere to put it except the answer column. Wrong credential became wrong answer. Wrong answer became 0 percent. And 0 percent became a fact I planned around.
What should have tipped me off was that the number never moved. Real evals are noisy. Skills improve a little, regress a little, flake on one case out of twenty. A metric that sits at exactly zero for months is not describing a model. It is describing something upstream that fails the same way every time. Flat numbers are plumbing numbers.
What broke
I treated 0 percent as a fact about my skills for months. I rewrote prompts and restructured workflows to move a number that was actually about a credential. The fix was deleting one line, and the score I had been chasing had never once been about the work.
02Two kinds of fallback
I build and run AI systems in several industries: a medical billing operation with a denial engine, an autonomous ad operator named Mia, a quoting engine for water-feature contractors, and the harness that evals my own agents. The same shapes keep turning up in all of them, and this bug taught me to sort fallbacks into two kinds.
A fallback in a compute path costs you quality. The big model times out, so you route to a smaller one. The cache misses, so you recompute. You get a worse answer or a slower one, but it is still an answer about the right subject. Degrading there is fine. Degrading there is the point.
A fallback in an identity path costs you the truth. Identity is the part of the system that says who is talking: which credential, which user, which account. When that part fails and something helpfully substitutes a different identity, nothing crashes. The run finishes and hands you a confident number about the wrong subject. My 0 percent was not a measurement of my skills. It was a measurement of a token, laundered through an eval harness until it looked like a measurement of my skills.
Key insight
A fallback in a compute path costs you quality. A fallback in an identity path costs you the truth, because the run still finishes and hands you a confident number about the wrong subject.
03The same shape in an approvals dashboard
A while later the same shape showed up in my ad operation. Mia is my autonomous advertising operator. Mia drafts the moves, and a human approves the risky ones through a dashboard behind Cloudflare Access, which stamps every request with a JWT that says who you are. The tempting fallback was obvious: if the JWT is missing, fall back to a session cookie so the dashboard keeps working.
I said no, and the eval bug is why. An approval attributed to a person who did not approve is worse than a login screen. A login screen costs you a minute. A false approval is a record that lies, and it lies in the one place you will later go to find out what happened. When identity fails, the only honest state is blocked. So the dashboard fails closed. No JWT, no page.
The result
The Mia approvals dashboard fails closed behind Cloudflare Access: no JWT, no page. Every approval in its history is attached to a person who actually clicked it.
The rule under both systems is the same one.
A 401 is not a wrong answer, and a missing JWT is not an approval.
An infrastructure failure is not a verdict about the work. My denial engine draws the line in the same place: its audit trail records which tools actually ran, never which tools the model claims it ran. It is the same line, drawn in three different industries.
04Ask who answered
The fix took five minutes. I deleted the fallback and made the script refuse to run without a real key. Auth errors now kill the run loudly instead of flowing into the score. The first honest eval was strange to watch, because the skills were fine. They had always been fine. Nobody had asked them anything.
What I took from it:
- 1Before you ask what a number means, ask who answered. Sometimes nobody did.
- 2Fall back on compute. Fail closed on identity.
- 3Keep error channels out of the answer column. A 401 is infrastructure, not a verdict.
- 4Distrust flat metrics. Real systems are noisy, and a number that never moves is usually measuring the plumbing.
If you run agents in production, go read your fallbacks. Most of them are fine. You are looking for the ones that swap who is talking, because those will hand you a confident number about the wrong subject, and you will believe it, like I did. The skills were never the problem. Nobody asked the model.