Zero Is Not a Score
The evals for my agent skills scored 0% for as long as I had records. Not low. Not noisy. Exactly zero, every skill, every run. And I believed it. For months I thought my skills were bad, because the number said so and the number never wavered. Then one night I actually read the harness. It had fallen back to the wrong auth token. Every call it made came back 401, and it quietly graded each one a failure. The skills never got a chance to fail on their own. I was not measuring them at all. I was reading a broken thermometer.
01Real weakness is jagged
Here is what took me too long to see. When a system is genuinely bad, it scores 40% one week and 60% the next. It passes the easy cases and trips over the hard ones. It has good days. Incompetence has texture, because an incompetent system is still in contact with the world, and the world varies.
A flat number has no texture. A flat number means the measurement stopped touching the thing being measured somewhere upstream, and what you are reading is the instrument's resting state. Doctors know this. A heart monitor drawing a perfectly straight line does not mean the patient is calm.
Only a broken thermometer writes the same number every time.
Key insight
A performance number with no variance is a reading of the instrument, not of the thing being measured.
02The same bug in three industries
I run systems in advertising, in healthcare billing, and in agent operations, and the same shape shows up in all of them. In advertising I found a dashboard figure that had been hardcoded for two years. Nobody questioned it, because it looked right, and it looked right because it never moved. In agent operations, an account-rotation bug in one of my pipelines overwrote every real error with the same generic message, "no active accounts," so for a while every distinct failure in that system looked identical. And in the denial-assessment engine I run for a medical-billing operation, an agreement metric came back at 44.7%, alarmingly low, until I noticed the two sides of the comparison were scored in different units. One side used an equivalence map; the other compared raw strings. Twenty-seven of the "disagreements" were the same operational decision written two ways. Rescored in units that meant something, the engine sat at 92.0% on classification and 90.7% on action against the golden set.
Three industries, one pattern. When a number is perfectly flat, perfectly round, or perfectly terrible, the odds you are looking at the system go down, and the odds you are looking at the pipe go way up.
03What a healthy number looks like
The week after I fixed that eval, the numbers got interesting. Interesting is the point. Mapped-action accuracy on the golden set went from 36.7% to 76.7% across four runs in one afternoon of prompt work. Classification went from 80% to 93.3%. Errors went from one to zero. The line wobbled, jumped, and climbed, and every point on it told me something I could act on.
That is what contact with reality looks like. A healthy metric breathes. So I have inverted my instincts. Steady numbers used to feel comforting and volatile ones used to feel alarming. Now the number that never moves is the one that keeps me up at night. I judge the autonomous operators I run the same way. When one of them turns in the same figure day after day, I do not ask what it is doing wrong. I ask what stopped being measured.
What broke
I believed a broken harness over my own work for as long as I had records. Zero felt like an answer, so I never read the grader. The fix was one auth token. The cost was every decision I made off that number.
04Grade the grader
The rule I run now is simple. Before I trust an eval, I make it prove it can pass. Feed it a case where the right answer is known and pinned, and watch it score a success. A harness that has never emitted a passing grade is not strict. It is dead. And dead harnesses grade perfectly, forever.
The result
The denial engine's eval now gates every ship at 90% against a golden set, and I trust that gate because I have watched it pass, fail, and change its mind as the system changed. It moves. That is how I know it is alive.
- 1Distrust any metric with no variance. Jagged is what real looks like.
- 2A flat zero is a flatline. Check the instrument's pulse before the patient's.
- 3Grade the grader. Run a known-good case through every eval and watch it pass.
- 4Never let a fallback swallow an error. One wrong token cost me months of belief.
- 5Perfectly round and perfectly stable deserve the same suspicion as perfectly bad.
The zero on my dashboard was never a grade. It was an instrument talking to itself. The numbers I trust now are the ones that move, because a number that moves is still touching the world. Most of us grade everything except the grader. Start there.