Underpowered Tests Lie
Last month I audited the statistics inside Mia, my ad-testing agent, and found a 20x error that had never broken anything. No exception, no failed assertion, nothing in the logs. The textbook shortcut said each arm of an A/B test needed 1,111 users. The exact formula, run at the 1.9 percent click-through rate ads actually get, said 22,278. Every test Mia had sized was running on a twentieth of the data it needed, and every one of them still announced a winner. I run AI systems in several industries, and the failures that cost the most all have this shape: nothing breaks, and the answer is wrong.
01Where the shortcut came from
The shortcut is Lehr's equation: sixteen over the effect size squared. It is in the textbooks because it is easy, and it is easy because it was derived where the math is friendliest, near coin-flip rates, where variance peaks and everything is symmetric. Ads do not live there. Ads live at 1.9 percent, where the effects worth catching are tiny in absolute terms and the samples you need to see them are huge. The shortcut knows none of this. It returns 1,111 and moves on.
The part that took me longest to accept is that the formula is not wrong. It works fine in the world it came from. The bug was mine. I carried it into a different world without re-deriving it.
02What an underpowered test says
You would think an underpowered test would fail. It cannot. There is no runtime check for statistical power. The test collects its too-small sample, computes a p-value, and hands down a verdict in the same confident voice it would use with twenty times the data. Sometimes the verdict is even right. That is what makes it dangerous. A crash gets fixed the same afternoon. A confident wrong answer gets acted on.
An underpowered test never says it does not know. It says B wins, and it says it with a straight face.
The audit found a second lie. Mia judged three-arm tests at raw p below 0.05, which sounds like a 5 percent false-positive rate. With three comparisons it is closer to 14 percent. One test in seven was crowning a winner that did not exist. Holm-Bonferroni fixes that in a few lines of code. Writing the lines took minutes. Suspecting I needed them took an audit nobody asked for.
What broke
The 20x sizing error and the 14 percent false-positive rate sat in production together, and the code ran green the whole time. Every signal I could see said the system worked. The only thing wrong was the answers.
03The same lie in healthcare
I caught this in ads, but I only recognized it because I had already paid for it in healthcare. Daniel, my denial-assessment agent, is graded against a golden set of 30 cases. When a prompt change moved classification accuracy from 80 to 93.3 percent, I did not believe it until the eval said the same thing four runs in a row. Thirty cases is a small sample. A single run there can lie as smoothly as an underpowered ad test, and it lies the same way: not with an error, with a number.
That is the cross-industry pattern. It has nothing to do with ads or insurance claims. It is about verdicts that sound the same whether or not they earned it. Ad tests, eval harnesses, dashboards: none of them has a voice for "not enough data." If you want that voice, you have to build it yourself.
The result
Mia now sizes every test with the exact unpooled formula at the observed base rate, refuses to declare a winner before that sample is met, and applies Holm-Bonferroni across arms. The fix took a day. Finding it took an audit nobody asked for.
04Re-derive at your base rate
Key insight
Every statistical rule of thumb is calibrated for someone else's base rate until you have re-derived it at yours.
That is the rule I run on now, in every industry I touch. Here is what it looks like in practice:
- 1Re-derive the formula at your own base rate before you trust it. The textbook derived it at theirs.
- 2Ask what the system says when it does not know. If the answer sounds the same as when it does, you have a liar, not a tool.
- 3Count your comparisons. Three arms at raw p below 0.05 is a 14 percent false-positive rate, not 5.
- 4Audit the judge, not just the work. The most expensive bugs live in the thing doing the grading.
- 5Treat green as a timestamp, not a verdict. It means the code ran, not that the answer is right.
The tests that fail loudly are the cheap ones. You see them, you fix them, you move on. The expensive ones succeed quietly at the wrong thing, for months, while everyone trusts them a little more each week. If your experiments run at base rates the textbook writers never pictured, this is the week to re-derive your formulas. And if you catch one of these lies in your own stack, write it up. The operators who actually check are rare, and we should find each other. The math was never the hard part. The hard part is doubting a number that looks right.