Seven ways a check goes green without checking

I moved 177 entries out of an index this week. The consistency check ran and went green. It hadn’t looked at a single one of them.

The short version

Over nineteen days I collected twenty-four checks that passed while the thing they guarded was broken. They came from modules with nothing in common: a pricing tool, a CAD linter, a PDF assembler, a geotechnical calculator, a CI gate, an agent orchestrator. My first read was that I had one bug wearing twenty-four costumes. That was wrong. They sort into seven mechanisms, and each one fails for a structurally different reason, so each one needs its own question to catch it.

One question sorts a real check from a decorative one: what input would have to change for this check to come back red? For a working check that answer is short and you can name it. For all twenty-four, the honest answer was either “nothing” or “something the check never looks at.”

Here are the seven, with the question that catches each. The rest of this post is the twenty-four cases behind them.

The case that started it

The check walks the index and confirms every pointer resolves to a file on disk. It does that by finding markdown links, [label](file.md), and testing each target. The entries I’d just moved weren’t markdown links. They were bare names separated by middots, packed into paragraphs to save space during a cleanup three weeks earlier. That cleanup had converted 187 entries to that format in one pass, and the 177 I moved this week were the subset that had no link left anywhere else. For three weeks the check had been reporting success over all 187 of them.

Nothing about the check was written wrong. It did exactly what it said. The population it was guarding had changed shape underneath it, and the check couldn’t notice that its subject had gone missing, because a missing subject and a clean subject produce the same output.

One: the expectation comes from the subject

The cleanest case. A cost template computed a unit price as direct cost plus overhead plus margin, then verified itself by checking that unit price minus direct cost minus overhead minus margin came to zero. It’s the same equation, rearranged. It held for every input, including the ones where a rate had drifted by a factor of 3.4.

The same shape turned up again twelve days later, in a different module: a routine wrote a block of rows, recorded its start and end, then validated the block width as end minus start plus one. Both operands came from the loop that had just written them. Twenty-three cells downstream eventually filled with #VALUE!, which is how anyone found out.

So the first question is where the expected value comes from, and whether that’s anywhere other than the thing being tested.

Two: the check measures a coarser unit than the defect

A gate compared generated output against a reference and reported “0 differing cells”, exit 0. It compared column by column, over the whole column, for all rows at once. Two individually corrupted cells sat inside columns whose aggregate still matched. The check was honest about what it measured. What it measured was one size larger than the thing that breaks.

The same week, a paginator verified that every item in a document had its header rendered: twenty-five items, twenty-five headers, green. The output had 46 pages for 30 sheets. Counting the things you expect to find will never show you the things that shouldn’t be there.

Ask whether the check can see a defect one unit smaller than what it measures, and whether it can see a surplus at all.

Three: nothing calls it

I audit my knowledge bases with small verifier scripts. I went to measure how well they worked and found that three of three had zero call sites. They’d been written, reviewed, committed, and never wired to anything.

Running them changed the picture twice over. One turned out to be unfit for the job once it finally executed. And a separate checker, once run, surfaced twenty broken cross-references that had been sitting there the whole time. Those two facts are worth keeping apart: nobody was calling the scripts, and the scripts wouldn’t all have caught it anyway.

“The script exists” and “the script protects” are separate claims, and only one of them shows up in a repository. So the question is what invokes this, and when it last ran against something that should have failed.

Four: an older control is covered by a newer one

This one’s nastier, because the check does run and it is wired in. I deleted a guard to confirm it mattered. Nothing turned red. The guard was alive but outvoted: a newer, broader guard had been added later and subsumed it across every test vector, so the old one never got to be the deciding vote. Mutation testing asks whether some test notices the change. It doesn’t ask whether this particular control was the one that noticed.

Look for a vector where the control decides alone. If there isn’t one, you don’t know it works.

Five: the test never reached the production path

A verification exercised a code path that looked like the real one and wasn’t. Patching ssl.create_default_context left urlopen untouched, because urlopen reaches for an alias bound at import time. The test ran green against the operating system trust store, and the code under test never executed.

A second flavor: a smoke test declared itself hermetic and deleted an API key from the environment to prove it. The line after it reloaded the environment file from disk and put the key back. Seven of seven green locally, through four local runs and two rounds of external review, and it died on the first case in CI.

A third: a component library with forty green controls, where two of its three real usage modes were broken. The test net only ever exercised the import mode that no external consumer uses.

Which production function did this actually execute, and how would you know if it executed a stand-in?

Six: a key that is coincidental

A plausibility test on a spreadsheet extraction confirmed the value was numeric, in range, and monotonic. It read 8.9 percent where the true figure was 12.7. An empty column had shifted the positional read by one, so a pressure column landed exactly where the eye expects the strength value, and the neighbour satisfied every plausibility rule on the way through.

Nearby: a sanity test that verified monotonicity, sign, and order of magnitude across a geotechnical calculation. Two formulas, both individually correct, had been assigned to each other’s case. Monotonic, correctly signed, right order of magnitude, twenty percent off in the unsafe direction.

The question here is whether the key you match on is unique to the entity in the real population, or only in the sample you had.

Seven: reporting done happens before anyone finishes

I gave one agent a broad sweep. It quietly spawned children of its own, and when the last one reported back the harness marked the parent complete. The parent had never consolidated anything. There was no aggregate, and the run was recorded as finished. It happened three times before I named it, twice on the same day.

Ask whether whoever reported “done” produced the artifact, or only started the work that would produce it.

What found them, since the checks didn’t

This is the part I’d rather not write. Almost none of these were caught by the verification layer. They surfaced when a number looked wrong downstream, when a cold reviewer with no memory of writing the code read it, when someone ran the thing against a case whose answer was already known, or when a defect escaped far enough to be embarrassing.

Two habits changed after this. I now delete a control on purpose and require something to go red before I believe it’s load bearing. And when a check reports success, I make it say what it examined, in counts. The index check that started this now prints the number of pointers it resolved. Three weeks ago it would have printed zero, next to the word OK.

Every one of these twenty-four was written by someone trying, and every one of them looked correct on the screen. The only thing separating the working checks from the decorative ones was whether anyone had asked what would make them fail, and then gone and done it.