I have an agent with memory. Not a vector database, just text files: an index of one-line summaries and a set of topic files that hold the actual detail. A hook runs on every prompt the human sends and decides, in milliseconds, whether to remind the agent to go read something. That decision is most of the system. Get it right and the agent shows up to a topic already knowing the relevant history. Get it wrong and you either drown the agent in false reminders or leave it blind at the one moment it needed a file.
I had already measured this hook once, a few weeks earlier. I trusted that measurement. I was wrong to.
The first measurement
I tagged sixty activations, every case where the hook fired and injected a reminder into a session. For each one I judged two things: was the match relevant, and did it fire when nothing was needed. The result was clear. Over-triggering was the dominant failure, about half of the sixty. Actual retrieval misses showed up in about five percent.
The verdict I wrote down: don’t build full-text search over the memory files. The keyword matcher is already finding the right file most of the times it fires. The problem is precision, not recall. Tune the matcher down, not up.
That was a real measurement, done honestly, and it had one limitation I noted at the time and then didn’t act on: it only looked at cases where the hook had something to say. Sixty activations, zero silences.
What that sample couldn’t contain
A hook that never fires produces no activation to tag. If the real failure mode is the hook staying quiet when it shouldn’t, a sample built entirely from activations will never contain a single instance of it, no matter how many you label or how carefully. I had measured the population I could see and written a verdict about the whole population.
Measuring the silence
So I went back and reconstructed every human prompt across two hundred sessions: 1,813 of them. Sixty-three percent triggered nothing at all.
I sampled sixty of those silent prompts and had them judged by a panel with access to the full memory corpus, index and topic files both, so the question wasn’t “did the hook do something reasonable” but “given everything that existed in memory at that point, was there a file this prompt needed and didn’t get.”
Fifty percent of the sampled silences were real failures. Half the times the hook said nothing, it should have said something.
Stratified further, one number mattered more than the rest: among silences that happened at the very start of a session, sixty-six percent needed a file. Two out of every three times a session opened cold and the hook stayed quiet, that silence was wrong.
One more detail from that panel narrows the fix a lot: zero of the confirmed failures were semantic misses, cases where the words in the prompt genuinely had no overlap with the file’s content. Every one of them was recoverable by a keyword that was already sitting in the text somewhere.
Where the matcher was actually looking
“Recoverable by keyword” and “the matcher still missed it” together point at one place: scope, not intelligence. The matcher only compared the prompt against the one-line index entries, never against the body of the topic files. A prompt naming a concrete component of some project wouldn’t match if that project’s index line happened to describe something else about it, a deadline, a status, a different detail entirely. The keyword was in the file. It just wasn’t in the twelve words of the index the matcher was allowed to read.
The cheap fix, tested and discarded
The obvious cheap move was to lower the bar: instead of requiring two keyword matches to fire, require one. I tested it against the same silence sample before adopting it, not after. It recovered about a third of the misses. It also fired on nearly everything else, reintroducing the exact over-triggering the first measurement had spent its whole verdict warning against. Measured, then thrown out, not argued out.
The second number, and the one that changed the design
The more useful finding came from splitting the same matcher by where in the session it fired.
| Position in the session | Precision |
|---|---|
| First prompt | 72% |
| Mid-conversation | 29% |
Same engine, same code, same thresholds. The number that changed was the prior, not the matcher. At session open there’s no context loaded yet, so the odds that a topic needs an outside file are genuinely high. Mid-conversation, the file in question was probably already read earlier in the same session, so firing again is usually noise, not signal.
The design that followed was narrow on purpose: run the aggressive matcher only on the first prompt of each session. Everywhere else, stay conservative.
The design that didn’t run
A cold read of the implementation found the design hadn’t shipped the way I’d written it. A boolean check meant to mark whether a session had already been seen sat on the right-hand side of an and, so it only ran when the cheap matcher had already come up empty. Sessions that opened with a match were never marked at all, and the “first prompt” branch ended up firing much later in the conversation instead. Seventy-four percent of activations were landing in the aggressive regime the design was built to avoid.
The matcher was fine. The design on top of it was fine. The line that routed between the two regimes had a bug that no amount of remeasuring precision would have surfaced, because it never touched what the matcher scored. It decided which branch ran before the matcher was called at all.
Measuring the design told me it was right. It didn’t tell me the code agreed with it.