Which local model should a coding agent run? Round 2: llama.cpp 0.9

The fleet, then and now

ON rows ran with generation caps 1024/8192 and OFF rows with the fleet defaults 128/512; the gap between those two is the first finding. Single-run pass, seed 42: every number here is one measurement where October’s were the mean of three, and the ± columns return with the 3-seed pass. The turbofcfusion-neo-coder row is record only — its quant is no longer on disk.

The mistake worth your time: caps

The first thing I re-measured collapsed. Agent scores fell to 58% and 64% where October said 96% and 97%, with the models answering empty in turn one. The receipts said why: the thinking chain burned the entire 512-token turn budget before any tool call, and the episode carried on with a hole in it. The October fleet ran ON rows at 1024/8192; this pass first ran them at the harness defaults of 128/512.

Nothing about the models had degraded; the harness conditions had. Re-run at the fair caps, the collapse inverted — UD-Q3 back to 96% agent, Heretic to 97% — and ON went from a tax to a win almost everywhere. If you benchmark agentic models, take this away: a thinking model measured with direct-answer caps measures the harness, not the model. The failure mode (empty replies, one turn, no error) looks exactly like a model regression unless the receipts record it. Mine did; that is the only reason this paragraph exists.

What moved with the engine

Four things stood out.

Thinking stopped being only a hard-code tool

In October, ON bought quality only on the hard tier. This time ON improved six of seven models. Heretic-on posts the fleet’s best code and agent, 97% on both, and MiniCPM-on ties the top of the board at 0.87 while keeping its 23ms turns — still the only model whose thinking is free.

Strata gained the most

0.70 to 0.78 on engine 0.1.57 with the fair caps; tools went 78 to 94% and agent 77 to 94%. The OFF row stayed at 0.80, so the engine work landed specifically in thinking mode.

OFF stayed put except tools

llama.cpp 0.9 kept every OFF global within ±0.03 of October, and most rows moved less than the run-to-run spread. The movement sat in tool calling: qwen36 went 94 to 100% with turns at 16ms, while Heretic trades five points of code for a 19ms faster turn.

The top is now a tie between modes

October’s ties were between OFF rows. This time MiniCPM-on 0.87 equals qwen36-off 0.87, so the pick is by use: qwen36-off for 16ms turns, MiniCPM-on for 84% code with receipts.

How much context does a model burn to pass?

The question a reader left on the first round: the agent loop re-sends the whole conversation every turn, so two models with the same pass rate can cost very differently in sent tokens. Until now the harness counted what the model emitted (out_tokens) and not what it read. This round every row carries context receipts — in_tokens per call for tools and code, tokens per agent session summed over the episode’s turns, average and median across the fleet’s 72 sessions.

Three things the receipts say.

Per-call burn is flat

Roughly 290-340 tokens in per tool call and 185-220 per code call everywhere, because those prompts are the same tasks. All the variance lives in the agent loop, where the conversation grows.

Thinking is a context tax too

Same model, same tasks, ON against OFF: +26% session tokens on qwen36, +57% on Heretic, +76% on UD-Q3, +117% on Strata, +151% on gemma. The latency table already showed one cost of thinking; the receipts show the second.

Quality per burned token is its own ranking

qwen36-off delivers the joint-top 0.87 at 4.8K tokens per session, and MiniCPM-on the same 0.87 at 6.2K — the cheap thinker. At the other end, gemma-on burns the most context in the fleet (11.1K average, 7.7K median) for 0.75, and Strata-on pays 10.1K for 0.78. The engine work bought its points expensively.

What I would tell you to do

Same as last time, with one amendment. Measure on your own hardware, read the exclusions, treat close rows as ties, and when a score moves, check the caps before you believe the model. An engine upgrade does not change what a model knows; it changes how the plumbing counts. The fleet still has no single winner: tools and cheap turns belong to qwen36-off, code and agent now belong to Heretic-on, speed-agnostic quality to MiniCPM, and long-form dispatch to Strata, which just closed most of the gap its first round left open.

Scoreboard and live report: jordimarsal.github.io/ggufbench. Code, tasks and receipts: github.com/jordimarsal/ggufbench. The first round, for context: which local model for coding agents.