The Quota Picked the Model
A line in my denial engine said the model was Sonnet. Every assessment that engine has ever served ran on Haiku. Nobody edited the config, nobody shipped the wrong build, and the line was true the day I wrote it. It just was not true any time the thing ran.
01The number was real, the name was not
The engine takes a denial, reads the codes and the payer language, and scores the odds of winning the appeal. It has a primary model and a fallback. Primary was Sonnet. When the primary comes back with a 429, the code drops to Haiku and keeps going, which is what you want at 2 AM CT with four hundred claims to clear before the morning.
On my key, Sonnet 429s. Not sometimes. Every time. So the fallback was never a fallback. It was the road. The primary was a door that had been locked since before I started measuring anything.
I found it while chasing something else, and then I had to go back and re-label the whole eval set.
The numbers were fine. They came from real runs on real denials, and I still trust them. What was wrong was the sentence I kept saying out loud about them.
What broke
For weeks I called the engine's accuracy a Sonnet number in rooms where the difference mattered, and every one of those numbers came from a model I had never once said the name of.
02The upgrade is the regression
Here is the part that should bother you more than the wrong name.
Think about who actually gets Sonnet. Not me. A client on a paid key with real headroom, whose requests never hit a limit, gets the primary on every call. The person with the better account is the only one running the model I never tested. Same commit, same container, same config file. The code did not change. The quota did.
We are trained to worry about the other direction. You picture the slow machine, the bad network, the throttled key, and you make the thing degrade gracefully. Nobody writes an eval for degrading upward.
A rate limit is a config value, so an eval score belongs to the account that produced it.
Key insight
An eval score is a fact about the account that ran it and not about the code, so the moment you hand that code to someone else the score quietly stops being true.
I pinned it. claude-haiku-4-5-20251001, spelled out with the date, because Haiku is what I actually validated. If I want Sonnet numbers I can go get Sonnet numbers, on a key where Sonnet answers.
The result
The engine now runs the model its numbers came from on every account, and each assessment records which model replied instead of which one was configured.
03The same bug in other rooms
Once you see this shape you cannot stop seeing it. I run systems in a few different industries, and it turned up in all of them inside one week.
An appeal deadline stored as 2026-10-13 rendered as October 12 on a laptop in Chicago. The date in the database was right. The reader's clock decided what it said. Anyone west of UTC lost a day on a deadline with money attached to it.
On the Medicare remote-monitoring billing platform, invite and password-reset mail never reached the testers. The code was correct. The sending account sat in a sandbox that only delivers to verified addresses, and mine was verified, so it worked perfectly for the one person checking whether it worked.
In the same denial engine, the local batch worker only woke up when something asked for the jobs list. A caller following the documented flow, post a batch and poll the URL you are handed, never touched that route. The job sat queued forever while the code was, technically, doing its job.
On the local box that serves open source models, which model answers depends on what fits in memory that night. The config has an opinion. Memory has the last word.
Different systems, one shape. The code is identical on every machine. The behavior gets picked by things nobody wrote down: a quota, a sandbox flag, a clock, a route somebody happened to hit.
04What I do now
- 1Pin the model. A fallback that fires every time is not a fallback, it is your default in a costume.
- 2Log what answered, not what was configured. The record should say Haiku because Haiku replied, not because a variable claimed so.
- 3Treat the account as an input. Quota, verified senders, region, sandbox state, clock. All of it is config you never wrote and will never see in a diff.
- 4Re-run the evals where the code is going to run. A score from my key is a fact about my key.
- 5Ask which way a difference fails. If the better environment is the untested one, the upgrade is the regression.
I used to think the code was the product and the environment was setup. It is closer to the other way around. The commit says what the system is supposed to do. The account it runs on decides what it does.
If you have a number you say out loud, go look at what actually answered. You can read every line and still not know. Some of it is written somewhere else.