Green Is a Timestamp
A test failed in my Medicare reimbursement engine on a day nobody touched the code. No commit, no deploy, no new data. I read the diff twice looking for the change that broke it, and there was no diff to read. What had changed was the date. The test had a hardcoded date in it, and the calendar had finally walked past it. The test was not wrong the day it was written. It was wrong now. That difference turned out to be the most useful thing I learned this year.
01The Test That Expired
The engine handles remote therapeutic monitoring reimbursement for Medicare. Some records can be backdated, but only within a 72-hour window, and a test checked that logic with a fixture date somebody typed in by hand. The day it was written, that date sat safely inside the window, and the test passed for the right reasons. Then every day after that moved it closer to the edge. Nothing in the code changed. The fixture aged past the window the way milk ages past its label, quietly and on schedule, and one morning the test went red. A false alarm: a correct system, a failing check.
The embarrassing part is how long I stared at the code before I looked at the fixture. When a test fails, you assume the code drifted. It took me a while to accept the other possibility: that the test drifted while the code stood still.
A green check is a receipt, not a promise, and receipts age.
02The Cron That Lied
The other failure came from the same fleet, on the ad-tech side, and it ran the opposite direction. An improvement loop runs there as a nightly cron. It grinds through its cycle and reports whether the night went well. To decide, it summed the return codes of its steps. The way it summed them, a timed-out step counted as OK. So when the loop started timing out, the report stayed green. For weeks.
What broke
For weeks the nightly improvement cron timed out and reported OK, because its status check summed return codes in a way that let a timeout read as success. Every morning the report was green. Every morning I believed it.
Put the two failures side by side. In healthcare, time turned a correct test into a false alarm. In ad tech, time turned a failing job into a false all-clear. Nobody edited either system. The only input was days passing.
03What Green Actually Means
Here is the mistake I had been making, and I think most people make it. I treated green as a state: the system works. But that is not what a passing check tells you. A passing check tells you that at one moment, under one set of assumptions, one specific question got the answer you hoped for. The moment is gone as soon as the check finishes. The assumptions keep drifting whether you touch the code or not. Some checks assume what day it is. Some assume a step that fails will say so. Some assume somebody is reading the report. All of those assumptions have half-lives.
Key insight
Green is a timestamp, not a state. A check verifies one moment under assumptions that expire, so the checks need an expiry audit the same as the code.
We audit code because we know it rots. We almost never audit verification, because a passing check looks like the one part of the system that is definitely fine. That is exactly backwards. The passing check is the part nobody is looking at.
04Expiry Dates for Checks
So now the fleet runs the same way in both industries.
- 1Treat every green as dated. The question is not whether the check is green but when green last meant something.
- 2Never hardcode now. A fixture with a wall-clock date in it is not a constant. It is a countdown.
- 3Make silence loud. Every cron reports to a heartbeat outside itself, so a job that goes quiet is an alarm, not an assumed success.
- 4Audit the auditors on a schedule. Fixtures get reviewed for wall-clock assumptions the way dependencies get reviewed for CVEs.
The result
Every cron in the fleet now carries an external heartbeat, and every test fixture gets audited for wall-clock assumptions. A silent job is an alarm by default, and no test ships with a date that can age out from under it.
None of this is clever. That is the point. The clever parts of my systems were never the problem. The problem was a date typed in by hand and a sum in a status script. The failures were cheap. The silence was expensive.
Code rots when you change it. Checks rot when you don't. If you run systems that are supposed to watch themselves, find your oldest green check and ask what it has verified lately. The answer is usually a date. The builders who ask that question need to find each other.