Grade the Grader: The Most Expensive Bug Ships in Your Eval

The most expensive bug I shipped this month was not in a model or a prompt. It was in the thing doing the grading. I found it three times, in three different industries, in the same month.

01The smartest model money can buy scored 0.241

I run a local-model training platform: five open-source students from the Qwen and DeepSeek families, one hardened 54-task exam, a promotion gate built on paired McNemar statistics. For weeks, none of the students could break through the gate. The obvious explanation was the comfortable one: small models, hard exam, keep training.

So I wired a frontier model, gpt-5.5, into the identical harness as a reference track. Same tasks. Same fixtures. Same scoring.

It scored 0.241.

Statistically indistinguishable from all five locals; paired McNemar p >= 0.31 on every single comparison. And 34 of the 54 tasks were failed by every model, frontier included.

There are exactly two readings of that number. Either frontier capability is fake, or the exam is broken. The exam was broken. Shared-vault evidence was bleeding between unrelated tasks, and a harness bug was silently truncating the reference model's outputs. The verdict I wrote into the experiment log that day: the constraint is task and fixture quality, not base-model capability.

I had spent the budget on the students. The bug was in the professor.

02The gate that could never pass

Same month, different industry. My denial-management engine reads healthcare claim denials and recommends the operational move: appeal, correct and resubmit, bill the patient, write off. Eval-driven prompt iteration against a golden set took mapped-action accuracy from 36.7% to 76.7% and classification from 80% to 93.3%, with hard errors down to zero. The system was getting sharp.

The ship gate was not. The gate required the tuned API to agree with a weaker raw agent, and it reported action agreement at 44.7%. Blocking. Alarming. Wrong.

When I audited the 150-case run, 27 of the "disagreements" were appeal level 1 versus peer-to-peer versus retro authorization. Same pursue family. Adjacent moves a human biller files in the same drawer. And the measurement bug underneath was small enough to embarrass: the API-versus-agent axis compared raw enum strings while the API-versus-golden axis used an equivalence map. One eval. Two rulers.

Scored consistently, and measured against golden truth instead of a weaker sibling, the tuned API sat at 92.0% classification and 90.7% action. On the genuine divergences it matched golden roughly 78% of the time; the agent managed 38%. I retired the gate and installed an API-versus-golden gate at a 90% threshold. The old one, in the words of my own log, measured the wrong thing and could never pass.

Requiring your best rater to agree with your worst one is not rigor. It is a ceiling you installed yourself.

03When you cannot fix the grader, become it

Third industry, same lesson, inverted. My Medicare remote-monitoring compliance platform lives in a domain where the grader is not mine to debug: federal contractors run screens against billing patterns, and the OIG publishes exactly what those screens are.

So the platform runs the auditor's own math against itself. Monthly, it computes the OIG contractor screens over its own claims and writes the results as immutable metric snapshots; a correction is a new version, never an update. Even a screen that does not apply gets recorded as N/A with a written reason, never silently omitted, so an auditor can see it was considered. The design goal fits in one sentence: alarm before the external contractor would.

That is what grading the grader looks like when the grader outranks you. Internalize it. Run it early. Freeze the receipts.

04The lessons

1. 1When every student fails the same 34 questions, stop grading students. Grade the questions.
2. 2A gate that can never pass is not a high bar. It is a broken instrument bolted across the road.
3. 3Never measure one comparison with two rulers. Every axis of an eval gets the same equivalence rules.
4. 4Disagreement inside the same operational family is agreement. Define equivalence before you count.
5. 5If the auditor's math is published, run it on yourself first, and make the snapshots immutable.

Three industries. A training loop for open-source models, a denial engine, a Medicare compliance platform. In all three, the highest-leverage engineering of the month was not the system being measured. It was the instrument doing the measuring. Your model is probably fine. Your prompt is probably fine. Go read your eval line by line, because the most expensive bug you are shipping right now is the one that decides what passing means.

These are not theories. They are line items from one operator's July. If you have caught your own grader lying to you, the builders need to find each other.