When Your Benchmark Finally Tells the Truth

Update — v0.1.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent failures into permanent standing rules — extract a candidate rule from a failure trajectory, replay-test it against history, promote only what survives. pip install cauterule gets you the full CLI, MCP server, export to 7 formats, and a bundled git rule pack. The field test report evaluates 4 models across 394 trajectories and is the source for every number in this article. Release notes · Changelog

---

CauterRule is an open-source sidecar that turns repeated agent failures into standing rules. It extracts candidate lessons from trajectories, replay-tests them against history, and is designed to promote only rules that look reusable and safe.

When we started field testing CauterRule, the early numbers looked bad enough that the story almost wrote itself. Local OMLX models looked weak. Parse failures were common. Some corpus slices would not run fully. It was tempting to conclude that the model layer was the main problem.

That was not the right conclusion.

The better conclusion, backed by the rerun data, was that the benchmark harness itself was manufacturing a meaningful part of the failure.

The before/after numbers were too large to ignore

The clearest evidence came from rerunning the same benchmark surfaces after parser, prompt, result-reset, and timestamp fixes.

Those are not cosmetic improvements. A benchmark that can only parse 3 of 10 or 3 of 11 golden rows is measuring something very different from a benchmark that cleanly parses 10 of 10.

To put that in perspective: the early benchmark was throwing away 73% of its own signal on the golden corpus. It was not measuring model quality. It was measuring parser fragility and then labeling the result as "model quality." If we had shipped a product roadmap based on that data, we would have optimized the wrong layer for months.

And it was not just golden. The raw synthetic corpus went from 124/145 to 145/145 processed rows. The raw sibling-repo corpus went from 0/10 to 10/10 — zero usable data to fully usable data. That is not a gradient improvement. That is the difference between a corpus you can reason about and a corpus that produces no signal at all.

What was actually broken

The field test exposed four harness failures. Each one looked like a model problem but was not.

This distinction matters because each of those failure modes points to a different fix path. If you treat all of them as "the model is bad," you will optimize the wrong layer.

A golden result set with 11 rows for a 10-trajectory corpus is not evidence of model instability. It is evidence that the reporting path allowed rerun contamination. Raw corpus failures caused by missing timestamps are not evidence that the LLM could not solve the task. They are evidence that the input contract was incomplete.

Each misdiagnosis has a real cost. Misdiagnose duplicate rows as model instability and you spend weeks tuning temperature for a file-append bug. Misdiagnose parse failure as a weak model and you upgrade to a more expensive model for a greedy JSON slicer. Misdiagnose blocked raw runs as runtime failure and you debug the execution environment for missing timestamps in the input data.

The cost of misdiagnosis is not just wasted time. It is that you build confidence in the wrong mental model of your own system.

The fixes changed the meaning of the benchmark

We made five specific changes:

1. reset results.jsonl and summary.json at run start
2. extract the first balanced JSON object instead of slicing from first { to last }
3. filter blank context items before validation
4. strengthen the prompt with a concrete filled JSON example
5. repair missing timestamps in blocked raw corpus assets

The report describes the transition in one sentence: the fixes moved the benchmark from "mostly noise" to "mostly signal." That is the single most important shift in the entire field test. Not because the numbers improved — but because the numbers started meaning something.

Here is a concrete example. Before the fixes, the local Llama model on golden produced 3 parseable rows out of 11. After the fixes, it produced 10 out of 10, and the result was: 8 pass, 2 fail. That 8P/2F is a real signal about the product's extraction quality. The earlier 3/11 was not a signal about anything except the parser.

The raw corpora told the same story at a different scale

The curated corpora are the cleanest signal, but the raw corpora are where volume lives. And the raw data told the same story at much larger scale.

Six raw corpora. Before the fixes, two were completely blocked and four were partially blocked. After the fixes, all six ran end to end. That is not "we improved some metrics." That is "we went from a benchmark that could not process its own data to one that could."

The report makes an observation about this that I think is underappreciated: a working runner plus a stable corpus is a compounding asset. Each future iteration starts from the accumulated knowledge of the last one, instead of rediscovering the same benchmark setup problems every time. That is the difference between a benchmark that gets better with use and one that decays.

The transition from plumbing to judgment

There is a moment in every field test where the character of the work changes. For CauterRule, that moment was the parser fix.

Before the fix, the work was plumbing: file contamination, JSON slicing, blank context items, missing timestamps. Each problem had a clear cause and a clear fix. The fixes were satisfying — numbers jumped, blocked corpora opened up, the benchmark started working.

After the fix, the work was judgment: is the trigger too broad? Is the directive specific enough? Does the replay logic correctly interpret the candidate? Does the product avoid extracting from cases where it should remain silent?

Plumbing fixes produce dramatic gains. Judgment-quality improvements require more careful scoring, better matcher design, improved negative-case handling, and human-calibrated evaluation. The project is leaving the easy gains behind and entering the more important work.

That is a healthy transition. But it is also a warning: the next set of improvements will not come from a one-line parser fix. They will come from harder, slower, more deliberate work on the product's decision quality.

What the benchmark started saying once it became honest

Once the harness stopped dominating the results, three conclusions became stable:

• the product could extract many candidate rules from positive or obvious failure trajectories
• the product was still much weaker on successes, failures/negative, and nearmiss
• the main unresolved problem had shifted from formatting to judgment quality

The report frames this as the project's biggest win — not a benchmark score, but the moment when "the remaining weak spots now point to the product, not to the test rig." Once that happened, the real product problem became visible for the first time.

And here is the quote from the report that stuck with me most:

A system that never produces useful output is obviously immature. A system that often produces useful-looking output, but still has weak safety boundaries, is more subtle and more dangerous. CauterRule is clearly in the second category.

The benchmark fixes did not make the product look good. They made the product look honest. And honest meant: good at the easy half, visibly weak on the hard half, and dangerous in exactly the way that matters for trust.

The engineering foundation held while the benchmark broke

One detail from the report that deserves more attention: while the benchmark harness was manufacturing failure, the deterministic foundation was solid.

• 844+ deterministic tests passing
• 104/104 Docker tests passing
• export/import validation passing
• redaction validation passing
• gold-family validation passing
• performance baselines within threshold

It would have been easy to dismiss the whole system as broken. It was not. The application core — the serialization, the CLI, the Docker environment, the export formats, the redaction engine — was reliable. The noise was concentrated in the benchmark layer, not the product layer. If the substrate had also been shaky, the field test would have been unrecoverable.

What I learned from this

A few things that I think generalize beyond CauterRule:

Your benchmark is a system too. It has its own bugs, its own input contracts, its own failure modes. If you do not test your benchmark, you are trusting an untested system to judge a tested one.

Signal loss is invisible until you measure it. We did not know we were losing 73% of golden signal until we fixed the parser and saw the numbers jump. Before that, the lost signal looked like "the model is bad." The failure was silent.

Failure bucketing changes your roadmap. If you lump all failures into "model problem," you optimize the model. If you split them into model / parser / corpus / scoring, you discover that most of your early failures are not model problems at all — and your roadmap shifts.

A plausible result from a broken benchmark is worse than no result. No result forces you to investigate. A plausible result from a broken harness gives you false confidence. The 3/11 golden rows were plausible enough to interpret. They were wrong enough to mislead.

Fix the benchmark before you optimize the model. If you optimize the model while the benchmark is broken, you are fitting to noise. Every model improvement you measure through a broken harness is suspect.

Raw breadth without replay clarity is not yet evidence. The raw corpora now all run end to end, but broad sets like raw/ci still generate many inconclusive results. The product can process them, but the benchmark does not yet turn that processing into consistently actionable quality signals.

Why this matters beyond one project

The broader lesson is not "always trust local models more." It is narrower and more useful:

never let your benchmark failure masquerade as a model failure.

If you are building an agent system, memory system, or any kind of model-evaluated workflow, split failures into buckets before you interpret them:

• model problem
• parser/tool problem
• corpus/input problem
• scoring/replay problem

If you do not, the benchmark can become the noisiest component in the system while still pretending to be the most authoritative one.

That is the part I would most want other engineers to argue with, test, or improve on. Not because it is abstract, but because once the benchmark becomes believable, the next round of product work becomes much more honest too.

---

CauterRule v0.1.0 is released. The full field test report — including the parser-fix before/after data, 844+ deterministic tests, 104 Docker tests, and model-by-model breakdowns — lives here. The repo is public. Install with pip install cauterule. Changelog · Release notes