We Could Have Shipped on Local Models Alone

Update — v0.1.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote. pip install cauterule gives you the full CLI, MCP server, 7 export formats, and a bundled git rule pack. The field test report evaluated 4 models (2 local OMLX, 2 cloud OpenRouter) across 394 trajectories. The local-model totals below — Llama 3.2B at 379 candidates and Qwen 4B at 373 — come directly from that report. Release notes · Changelog

---

CauterRule is an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization.

One of the easiest mistakes to make in AI engineering is asking the wrong question. We started our field test with this one:

Are small local models good enough?

By the end of the work, the more useful question looked different:

What are small local models good enough for?

And after the cloud runs came back, a third question surfaced that I did not expect to be able to answer affirmatively:

If I only had local models, could I still ship this product with the same confidence?

The answer turned out to be yes. Not because local models are as strong as the best cloud model — they are not. But because the thing that blocks shipping confidence is not the model. It is the replay engine and the safety corpora, and cloud models do not fix those. More on that below.

Two models did not even make it to the benchmark

Before the comparison could start, two candidate models were cut. Qwen3.5-4B-4bit was too slow for practical batch work — a model that takes minutes per trajectory is not one you can run repeatedly across 394 trajectories. google/gemini-2.0-flash-001 was unavailable on the tested OpenRouter tier. The endpoint simply was not there.

That is an operational detail that benchmark papers often skip. Two out of six candidates were eliminated before a single extraction ran. Model selection needs an operational screen — speed, availability, cost — before you invest in a full corpus sweep. Skip that screen and you waste real testing time on models that were never going to work in your workflow.

The field test report puts this bluntly: not every model failure is a product signal. Some are simply deployment or availability problems. The report needs to separate those cleanly so they do not pollute the product conclusions.

The post-fix local results were better than the early benchmark implied

Most of what the early benchmark called "local model failures" were not model failures at all. They were benchmark failures. The field test report puts this bluntly: before the fixes, both local models showed severe parse failure rates on curated corpora. After the fixes, parse failures dropped to near-zero on most curated corpora. The models did not change. The benchmark did.

That is six separate harness defects — none of them in the models — that were collectively making the local models look worse than they were. When the early benchmark said "small local models are borderline unusable," the honest reading was "small local models are borderline unusable through this broken benchmark." Fix the benchmark and the same models produce 379 and 373 candidates across 13 corpus types.

This is the part of the story that is easy to miss if you only look at the final numbers. The early narrative was not "local models have some weaknesses." It was "local models are mostly failing." That narrative was wrong, and the reason it was wrong is that the evaluation harness was failing the models, not the other way around.

After the fixes, the OMLX models became much more meaningful engineering tools. The cleanest curated comparison:

Not a story of "small local models solved the problem." But also not a story of "small local models were worthless." The full benchmark totals reinforce that.

379 candidates from a 3B model running locally on a laptop. 373 from a 4B model. Those are large enough run volumes to support real engineering decisions, not just gut feelings.

But the raw pass rate only tells part of the story. To see how local models actually stack up, you have to put them next to the cloud baselines from the same field test:

Read that table carefully. Local Qwen 4B — a 4B model running on a laptop — outperformed openai/gpt-4o-mini on total passes (93 vs 77) and on raw pass rate (24.9% vs 19.5%). Local Llama 3.2B beat GPT-4o-mini on the golden corpus (8 pass vs 6 pass). This is not "small local models earned a place as a regression tool." This is "a 4B local model beat a paid cloud baseline head-to-head on total pass count."

The cloud Llama 3.1 8B is still the strongest model in the field test, and the safety corpora remain weak across the board. But the premise that local models are clearly inferior to cheap cloud baselines is not supported by this data. On some corpora, the local models are better.

And here is the point that matters most for anyone deciding whether to ship: the cloud runs did not move the shipping decision. The field test report says it plainly — "better models improve extraction quality, but they do not erase the product's safety and replay problems." The strongest cloud model still fails on successes, failures/negative, and nearmiss. The replay engine is still the named #1 bottleneck. The safety problem lives in the product, not in the model. So if your shipping gate is "do we have enough confidence in the safety and replay layer," cloud models do not get you over that line any more than local models do. They get you a cleaner extraction ceiling, not a safer product.

That is why the honest answer to "could I ship with local models only?" is yes. You would ship with the same unresolved safety and replay gaps either way. The cloud model does not close them.

What the pass counts are actually measuring

There is one more thing the headline numbers do not show, and it matters for how you read every table in this article. The pass / inconclusive / fail verdict comes from the replay engine, not from a human judge. And the field test report names that replay engine as the single biggest bottleneck in the whole system:

Replay and matcher calibration is now the highest-value engineering target.

The matcher uses substring and token-overlap heuristics. When it cannot decide whether a rule fires against a reference trajectory, it returns inconclusive — not fail. That means a large slice of the inconclusive bucket is not a model failure. It is a replay-engine limitation. A specific, reasonable rule can land in inconclusive purely because the matcher is too weak to verify it.

So the pass counts in the tables above are closer to a floor on model usefulness than a ceiling. The true model-only pass rate sits somewhere between the reported pass count and pass-plus-inconclusive. For Qwen 4B, that range is 24.9%–81.0% of candidates. For Llama 3.2B, it is 19.0%–68.9%.

That is a wide band, and it is not honest to claim all inconclusives as hidden passes. Some of them are real model weaknesses — over-broad triggers like "when a command fails" instead of "when git push fails with non-fast-forward", and over-triggering on the successes corpus where the model should have stayed silent. Those are model problems, not matcher problems.

But it is also not honest to present the raw pass count as the model's ceiling. The report is explicit: the replay engine is the named bottleneck, and better models did not erase the inconclusive problem. So when you read that Local Qwen 4B passed 93 of 373 candidates, the honest interpretation is "at least 93, and some unknown fraction of the 209 inconclusives are also correct extractions that the matcher could not verify."

The raw corpus split: Llama wins breadth, Qwen wins depth

The curated corpora showed Llama as the better default. The raw corpora told a more nuanced story.

Look at raw/synthetic: Qwen produced 36 pass vs Llama's 18 pass — double the extracted rules. But Llama had 26 hard fails vs Qwen's 22. And both produced a large number of inconclusives (101 vs 87).

Then raw/ci: Llama produced 8 pass / 49 inconclusive / 52 fail. Qwen produced 9 pass / 90 inconclusive / 11 fail. Qwen had fewer hard failures but far more inconclusives. That is a different quality profile — Qwen is more cautious, Llama is more decisive but also more wrong.

This is why raw pass count alone is not enough to pick a default local model. The pattern of pass / inconclusive / fail matters. High pass + high fail means decisive but noisy — you get more signal but more garbage. Moderate pass + high inconclusive + low fail means cautious — less garbage but less signal. For a local regression tool, you want the decisive model — false positives are cheaper because you can review them. For a trust benchmark, you want the cautious one — false positives are expensive because they can be promoted.

What local models were actually good for

After the fixes, the local models were useful for:

• broad internal corpus sweeps
• low-cost repeated reruns
• prompt and parser regression checks
• local smoke tests for the extraction loop
• cheap model-to-model comparisons

That is already a meaningful role. And once you look at the local-vs-local comparison in a practical way, the split becomes clearer:

This is why I do not think raw pass count alone is enough to pick the default local model. Engineering usefulness includes speed, rerun friction, and how easy a model is to keep in the loop.

Where local models were still clearly limited

The same results also make the limits obvious. The weakest areas remained the same ones that matter most for trust: successes, failures/negative, and nearmiss.

That means the local models are still not strong enough to be the only evidence behind public quality claims or autonomous promotion decisions. But neither are the cloud models — the report's central finding is that even meta-llama/llama-3.1-8b-instruct does not solve the safety corpora. So the honest caveat is not "local models are only good for regression." It is "no model in this field test is sufficient for trust on its own, and the safety problem lives in the product, not in the model." Local models are competitive with cheap cloud baselines on extraction quality; what they all still lack together is safe rule selection and replay trust.

An observation that surprised me

I expected the larger local model (Qwen 4B) to clearly outperform the smaller one (Llama 3.2B). It did not. On the curated corpora, Llama 3.2B actually produced more golden passes (8 vs 3) and was more decisive on failures/positive — 15 pass / 3 inconclusive / 12 fail, versus Qwen's 15 pass / 9 inconclusive / 6 fail. By raw pass-to-fail ratio, Qwen actually wins that corpus (15:6 ≈ 2.5 vs Llama's 15:12 ≈ 1.25). But Llama produced fewer inconclusives, which means it committed to a verdict more often. Qwen caught up on breadth — especially on raw/synthetic where it doubled Llama's pass count — but it did not translate that into better curated precision.

That suggests something I did not expect: for small local models, raw capability and extraction precision may not correlate the way you'd assume. A 3B model with good instruction-following can outperform a 4B model on specific tasks, even when the 4B model processes more data. The bottleneck at this scale is not parameters. It is instruction adherence and format stability.

What I learned from this

First impressions from a broken benchmark are worse than no benchmark at all. The early data made local models look borderline unusable. After the fixes, they produced hundreds of candidates across 13 corpus types. The models did not change. The benchmark did. If we had trusted the early data, we would have written off local models entirely and spent money on cloud APIs for every regression run.

Operational screening belongs before benchmarking, not after. We lost time on Qwen3.5-4B (too slow) and Gemini Flash (unavailable). Both failures were predictable with a 5-minute operational check. A full corpus sweep across 394 trajectories is expensive. Do the cheap screening first.

Pass count is not the same as decision quality. Qwen 4B had more total passes (93 vs 72) but also more inconclusives (209 vs 189). Llama 3.2B had more hard fails (118 vs 71). Which model is "better" depends on what you are using it for — a regression tool wants decisiveness, a trust benchmark wants caution.

Role separation beats model replacement. The temptation when a model is weak is to replace it. The better move is to assign it a role it can handle. But note what the data actually showed: a 4B local model beat a paid cloud baseline on total pass count. So "local for iteration, cloud for quality" is true, but it is not true because local models are weak. It is true because cloud models give you a cleaner quality ceiling and wider model choice, not because local models cannot compete on extraction. The cheapest useful cloud model (meta-llama/llama-3.1-8b-instruct) still beat every local model tested — but gpt-4o-mini did not.

Open questions

• Would a 7B or 8B local model close the gap on successes and nearmiss, or does the safety problem live in the product regardless of model size? The cloud results in the next article suggest the latter.
• Is the Llama 3.2B advantage on golden stable across reruns, or is it within noise? We ran each model once per corpus. More reruns would tell us whether 8P/2F is reliable or lucky.
• The raw/synthetic split (Qwen 36P vs Llama 18P) is large enough to matter. Is Qwen genuinely better at breadth, or is it over-triggering on raw data? The inconclusive count (87 vs 101) suggests Qwen is extracting more but also hedging more.

The broader lesson

Local models were not good enough to end the evaluation conversation. But they were much better than their first benchmark impression — good enough, in fact, to beat a paid cloud baseline on total pass count, and good enough to ship the product on. The lesson is not that local models are a cheap fallback. It is that the gap between "cheap local" and "cheap cloud" is smaller than the benchmark infrastructure made it look, and the thing that blocks shipping confidence — the replay engine and the safety corpora — is a product problem that cloud models do not fix. If I only had local models, I could have shipped CauterRule with the same degree of confidence the cloud models gave. Which is to say: the same confidence, and the same unresolved gaps.

---

CauterRule v0.1.0 is released. The full model-by-model breakdown — including the curated corpus comparison table, raw corpus results, and model ranking — is in the field test report. The repo is public. Install with pip install cauterule. Changelog · Release notes