Every audit deleted the audit before it

For four months now I have been running a monthly audit over the hook system that keeps my agent’s corrections from evaporating between sessions. It reads the fire log, labels each fire with an outcome, renders a dashboard, and writes a report. It has caught real problems. In July it caught a classifier that had been dead for weeks behind a passing green dashboard.

Last week it caught something better: itself.

The audit was deleting the history it was supposed to be accumulating. Not corrupting it, not failing to write it — deleting labels that were already there, on every run, and then reporting a number computed from the rubble as if it were a trend.

Two producers, one field

Outcome labelling works like this. A hook fires. At the moment it fires, nobody knows whether it did anything useful — that is the whole difficulty of measuring this kind of system, and I have written about it before. So the label is assigned afterwards, by a batch job with a per-hook heuristic, into a field called outcome.

For most hooks a heuristic is enough. For one of them it is not. That hook fires on every prompt and recommends reading specific memory files; whether the agent actually went and read them is not visible in the fire record at all. It is visible in the session transcript. So that hook has its own marker: a second job that opens the transcript, looks for the reads, and writes a real label — error_prevented if the agent complied, error_despite_fire if it ignored the recommendation and then made the mistake the file would have prevented.

Two producers, writing the same field. The general batch job runs first. The specialized marker runs after it, in the same pipeline, and the pipeline runs monthly.

The general job has a skip rule, so it does not stomp on work that is already done:

if fire.get("outcome") and fire.get("outcome") != "unknown" and fire.get("reviewed_by_human"):
    # already marked, leave it alone
    continue

Read that condition carefully, because I didn’t, for three months. It preserves an existing outcome only if the fire also carries reviewed_by_human. The specialized marker doesn’t set that flag. It identifies its work through a different field, outcome_marked_by, which is the obvious thing to do and which the skip rule doesn’t consult.

So every month, the general job looked at the specialized marker’s labels, decided they were not protected, and re-classified them. And its heuristic for that particular hook returns — correctly, by design — unknown, because a batch heuristic genuinely can’t judge those fires. That is the whole reason the specialized marker exists.

The result was not a wrong label. It was a deleted one.

Thirty-two seconds

I can put a clock on it, because the pipeline takes a backup before each destructive step, and the two backups from the September run bracket the deletion.

The backup written immediately before the general job: 794 fires carrying a known outcome from the specialized marker. Broken down by month — 23 from June, 770 from July, 1 from August.

The backup written by the specialized marker thirty-two seconds later, before its own pass: zero. Every one of those 794 rows now said auto_classify_rules, and every outcome said unknown.

July was the expensive one. 333 error_prevented and 437 error_despite_fire — real labels, each one produced by opening a real transcript and checking what the agent actually did — reduced to unknown in a single step.

Then the specialized marker ran and rebuilt what it could. It could only rebuild fires whose transcript was still on disk. Transcripts age out. The count of fires it had to skip for a missing transcript went from 514 to 1,285 in one month.

So the cycle was: destroy the labels, rebuild the recent ones, report a rate computed from the survivors. Every month. The number was always about the freshest month, because the freshest month was the only one with transcripts. I had been reading it as a series.

The part that still bothers me

Here is the detail I keep coming back to. I went to check how much the skip rule was protecting, because it clearly protected something — human review is a legitimate thing to guard.

Across the live log and both backups — 14,176 rows, 9,022 rows, 13,864 rows — the number of fires with reviewed_by_human == True is zero.

Nobody has ever marked an outcome by hand. Not once, in the four and a half months the log has existed. The condition was guarding an empty set. It preserved nothing and destroyed everything else, and it did that for three consecutive audits without a single symptom.

That reframed the fix for me. I had been thinking of it as a trade-off — loosen the guard, accept some risk of clobbering a human decision. There was no trade-off. There were no human decisions to clobber.

Why nothing looked wrong

A deletion like this has no error surface. That is what makes it worth writing down.

The row survives. Nothing is missing from the file, no count drops to zero, no exception is raised, no schema is violated. The row simply says unknown where it used to say something, and unknown is a completely ordinary value that most rows legitimately have. The monthly totals still moved in plausible directions. The dashboard rendered. The health checks passed — and there are health checks, and they are not decorative; one of them is what caught the dead classifier in July.

The reported incompliance rate for that hook came out at 77.4% for August, up from 56.8% for July. I wrote that comparison down as a twenty-point deterioration and started thinking about why the hook had gotten worse. It hadn’t necessarily gotten worse. The two numbers were computed over populations that one of my own pipeline steps had been reshaping between measurements.

The only artifact that showed the deletion was the backup that the deleting step wrote about itself, one second before doing it. I want to be precise about the luck involved: I didn’t find this by being careful. I found it because a routine step had a --backup flag turned on and I happened to diff two files that a normal run wouldn’t ever compare.

Precedence, not exceptions

The fix is small and the shape of it matters more than the code.

The wrong fix is to special-case the specialized marker — add its name to the skip condition and move on. That closes this instance and guarantees the next marker I write arrives with the identical bug, because nothing in the system would say so.

The rule is that outcome writes have a provenance, and provenance is ranked:

manual  >  specialized  >  auto  >  none

A producer may overwrite an outcome only from a strictly higher rank, or rewrite its own — a legitimate re-run. An absent or unknown outcome protects nothing, so the first writer always wins the empty field. That is the entire rule, and it lives in one function that every producer consults, instead of being implied by a boolean flag that one of them happens to set.

The invariant it buys is worth stating separately, because it is the thing I didn’t have and should have:

Two consecutive runs of the pipeline, with no new evidence, leave every historical outcome byte-identical.

That is a test. It is not a performance property or a nice-to-have; it is the property that separates a pipeline that accumulates a series from one that overwrites it. Run yours twice and diff. If the second run changes history, you don’t have a series — you have the most recent month, dressed up as a trend.

Before the fix, running the general job twice over the real log destroyed 1,361 labels. After, it destroys none. I didn’t verify that by reasoning about the condition; I copied the live log and ran the job twice against the copy.

What came back

The July labels were still recoverable, because the backups were still on disk. The restore merges by event id — never by line position, since the live log grows while you work — and takes the higher-ranked outcome for each row.

746 fires restored, plus 48 that the marker had already correctly rebuilt: 794, which is exactly what the pre-deletion backup held. Zero invented ids, zero lost, zero duplicated.

For the first time the series exists:

The August jump is real and now rests on comparable populations, which the earlier version of that same comparison didn’t.

One honest caveat that survives the fix: about half of that hook’s fires are false positives — system events and notifications where no recall was ever needed, so “ignoring the recommendation” is the correct behaviour rather than a failure. The rate above is not “the agent ignores this hook eight times out of ten”. Separating the two requires hand-reviewing a sample, which I have now put off for three months and which is the next thing I owe this number.

The general shape

I think the transferable lesson is narrow but sharp, and it is not really about hooks.

A pipeline step that recomputes a field is a delete-and-rewrite of that field. If two producers can write it, and their skip rule is not the same rule, the one that runs first is silently authoritative and the one that runs second is decoration. The failure is invisible by construction: recomputation is the normal operation of the step, so there is no anomaly to detect, no log line to grep, and no alert to fire.

Three things I would now do by default anywhere a field has more than one writer:

Make provenance a value, not an inference. Stamp who wrote it on the row. Don’t derive authority from a flag that one producer happens to set, because the next producer won’t set it, and nothing will tell you.

Test idempotence on data, not on logic. The condition read fine to me for three months. Running the job twice over a copy of the real log took about a minute and was unambiguous.

Diff your backups occasionally, not just when restoring. The backups were the only witness here. They had been sitting there since July, containing the whole story, waiting for someone to compare two of them.

The precedence rule is now in fscars, the open-source version of this scar system, which carried the identical defect — same condition, same skip rule, written from the same instinct a year apart. OutcomeMarker there now ranks writes and refuses to let a batch pass overwrite a specialized one.

One more turn of the same screw

I put that fix through a cold review before merging it: a fresh agent, no memory of writing the code, told to run the tests itself and try to break the claims rather than confirm them.

It rejected the first version. The docs I had just written said a producer may overwrite an outcome only from a strictly higher rank. The apply() function — the only path that actually writes — never checked. It validated the name of the provenance and then wrote unconditionally. So the path I had documented for a specialized marker would overwrite a human label.

The same defect. One rank up. In the fix for it, written by someone who had spent the day thinking about nothing else.

A second round found five more, and the one that stings is the smallest: a test I added to pin the version number imported tomllib, which is standard library only from Python 3.11. The package supports 3.10. That single import would have broken three of the nine CI jobs — a test written to close an audit finding, breaking the gate that catches audit findings.

I don’t think this reflects unusual carelessness, and that’s the uncomfortable part. Each of those defects is the same shape as the original: a rule stated in one place and enforced in another, with nothing tying the two together. Writing the rule down felt like implementing it. It reads like implementing it. The gap only becomes visible when something that did not write the rule goes looking for where it is enforced.

Which is the whole argument for the layer, restated at a level I did not expect to be standing on: the check has to be outside the thing being checked. I built a system on that premise, and I still could not review my own patch.

The audit found the audit, and a reviewer found the fix. I would rather all three had been tests.