Epistemology for developers: who checks the tests

Fourteen hundred tests green, eighty-seven per cent coverage, and Thursday’s deploy left checkout accepting orders without charging for them. The postmortem turned up the test covering that exact path. It passed. It had been passing for eleven months.

It had been written from the code, not from the requirement. It asserted the same thing the function did, in different words, which is why it could never contradict it. The green wasn’t a check. It was an echo.

That suite wasn’t measuring the system. It was measuring itself.

The question nobody asks in the retro

A test decides whether the code is right. The uncomfortable question comes next: who decides whether the test is right?

The usual answer, if you press for it, is circular. The code is right because the tests pass. The tests are right because the code works in production. Each half rests on the other and the whole thing rests on nothing.

The alternative isn’t free either. You can put something above the tests to watch them — mandatory review, a coverage floor, an assertion linter — but then you need something to guarantee that the review reviews, that coverage measures what it claims, that the linter rule was the right one. Every level of oversight needs its own. There is no last one.

Either the criterion justifies itself, or it leans on another that needs a third. A circle or an endless staircase.

A two-thousand-year-old problem

It isn’t a problem of our trade. It’s the problem, and it was stated a long time ago.

The Greek sceptics called it, without ornament, the problem of the criterion. To settle a dispute you need a measure both sides accept. But when the dispute is precisely about which measure to use, there’s nothing to stand on: justify it with another measure and you open a series with no end; justify it with itself and you’ve assumed what was in question. Sextus Empiricus wrote it down with a textbook coldness that is still unpleasant to read.

Against that argument stood the Stoics, who held something very close to what a team with a good suite holds: that some impressions carry their own guarantee, so clear and so well stamped that they could not have come from something non-existent. Zeno taught it with his hand. Open, the impression arriving. Fingers folded, assent. The closed fist, comprehension. The other hand gripping that fist, knowledge.

The whole sceptical offensive consisted in showing that for any such impression you can construct another, indistinguishable and false. Not that we’re always deceived. That there’s no mark on the impression telling you which is which.

Translated: there is no green that carries its own guarantee. For every suite that passes there is an indistinguishable suite that also passes and covers a broken system. The one we had that Thursday was that one.

The sceptics didn’t stop living

Here’s the turn that matters, because the argument above, taken raw, leads somewhere useless: if nothing guarantees anything, never deploy. Nobody works that way, and the sceptics didn’t live that way either.

They distinguished two things our trade confuses constantly. One is the criterion of truth, the thing that would guarantee something is so. That’s the one they demolished. The other is the practical criterion, the guide you conduct yourself by: what appears, what the body asks for, local custom, and the training of a craft. That one they accepted without trouble, on one condition — not taking it for true.

A test suite is the second kind. It’s a very good guide to conduct, distilled from the craft, telling you where to look and in what order to move. It is not proof that the system is right. It works well while it’s used as what it is, and it fails the day somebody promotes it to a guarantee and stops looking.

Almost all the damage I’ve seen comes from that promotion, not from a lack of tests.

Carneades’ three grades

Carneades, who ran the Academy and never wrote a line, left behind the most usable thing in all of this: a scale. Facing an impression you can’t guarantee, you can still grade how much weight you put on it. He distinguished three levels, and his example was examining candidates for public office.

The first: the impression is convincing on its own. You look at it and it persuades you.

The second: on top of that, nothing around it contradicts it. You’ve looked about and everything agrees.

The third: on top of that, you’ve examined it thoroughly, point by point, hunting for where it might fail.

Worth saying that his word didn’t mean “probable”, though it was translated that way for centuries and the mistake still circulates. It meant convincing. It isn’t a measure of how close something is to the truth; it’s a measure of how much it has persuaded you. The difference isn’t small: the first claims to talk about the world, the second admits it is talking about the observer.

On tests, the scale reads itself.

Grade one: the test passes. Convincing on its own. Most suites in the world live here and stay here.

Grade two: nothing else contradicts it. The types line up, the neighbouring test is still green, the consumer of that API hasn’t complained, staging has been behaving for a week. None of those proves anything on its own. Together, and being independent of each other, they’re worth considerably more than the first grade.

Grade three: somebody tried to knock it down on purpose. Mutating the code to check the test actually goes red. A real load case. A hostile review. A test written by someone who didn’t write the function, from the requirement, without opening the implementation.

Grade three is expensive. Which is why it doesn’t go everywhere: it goes where failure is paid in money, in data or in trust. Everywhere else, grade two is already a reasonable luxury.

What I changed after that Thursday

Four things, and none of them is buying more coverage.

• A test written from the code is not evidence. It’s a certified copy of the implementation. Write it from the requirement, before or apart from the code, or have someone else write it. If it comes from the same place as the code, it inherits its mistakes whole.
• See it fail before you see it pass. A test that has never been red hasn’t even shown it’s capable of failing. It’s the cheapest check there is and the one most often skipped.
• Look for a different kind of evidence. Faced with something that matters, the question isn’t “does the test pass?” but “what else, not depending on this, points the same way?”. Two proofs that share an assumption are one proof said twice.
• Name the grade. When I close a PR I know whether I’m at grade one or grade two, and I say so. Not to look rigorous, but because whoever deploys deserves to know what they’re standing on.

Doubt costs, so it gets budgeted

The conclusion isn’t to doubt more. Doubt has a price and the budget is finite: every hour spent securing the edge is an hour not spent on the core.

What changes is where you put it. There’s a part of the system where an error gets fixed next Tuesday and nothing happened, and grade one is plenty there. And there’s another where an error surfaces six months later while reading a complaint, and there not even grade three is excessive.

In the bug that doesn’t exist the subject was the gap between what the system does and what the system says about itself. This is the same gap, seen before anyone shows up to live in it.

The sceptics didn’t win the argument, partly because they weren’t trying to. What they left is more useful than a win: the habit of asking, in front of anything that gives you confidence, where its own confidence comes from. A green build doesn’t answer that. It only postpones it until Thursday.