Tag: evals

All the articles with the tag "evals".

• Ninety days of scars

  10 Jul, 2026

  Ninety days ago I started enforcing corrections outside the model. This week I pulled the first real metrics, and it is a mixed result. The busiest hook gets ignored 62% of the time, two of my own tools disagreed about the same number, and the dashboard stayed green while a whole layer was dead.

• How do you know a correction held? Instrumenting an agent in production

  9 Jun, 2026

  Functional scars make a correction persist. They don't tell you whether the system as a whole is getting better. So I instrumented every session — deterministic, zero-token, never blocking — and let a monthly pass turn the evidence into mechanism changes. The first thing the data caught was me.