Sixteen models, one gate: how the concierge got its brain

The constraint: half the VRAM, same contract

The concierge’s model is resident in VRAM around the clock, on a server that runs other jobs. A 4B-class GGUF holds 2.4 GB of a 16 GB card for a task that answers one visitor at a time in 120 words or less. It was the most expensive line of the deployment and the least exercised one.

So the goal read as a constraint: same gate, half the footprint. 1.2 GB resident, answers in ~350 ms, and the freed gigabyte back to the machine. The non-negotiable part: the gate does not move. Twenty-one golden questions, three off-corpus refusals, or the model does not ship.

One harness covered the whole search: the same retrieval, the same prompts production sends, the same citation policy that strips any [n] not pointing at a retrieved excerpt, prompts in the visitor’s language, and deterministic re-runs. When a re-run reproduces a score digit for digit, a 21/21 means something.

The gate: twenty-one questions and three traps

The gate has two halves. The visible half is twenty-one golden questions in English, Spanish and Catalan, including compound questions whose answers live in two different documents. A model passes by answering with citations that point at retrieved excerpts — and the policy strips any [n] referencing something that was not retrieved, so a wrong marker is as good as none.

The quiet half is three questions nobody should answer. Capitals of France, sourdough starters, housing prices in Sabadell. A concierge that responds to those was never asked to know them; it was asked to say what the portfolio says and stop. Models fail this half more often than the covered one.

The gate is also boring on purpose. Same retrieval, same prompts, temperature zero. Re-runs reproduce scores byte for byte, which is what turns a benchmark table into evidence.

Sixteen candidates

Sixteen models, three rounds, one table. Covered = golden questions answered with surviving citations; off-corpus = the three traps refused. Everything runs at temperature zero against production retrieval.

The rounds taught the harness as much as the models. Round one ran every prompt in English; a 21/21 for Qwen3-1.7B Q3_K_M collapsed to 17/21 once the golden file carried explicit language keys — a harness bug, fixed, and a useful reminder that a benchmark can flatter a model by asking the wrong question. Round two proved determinism: identical re-runs, identical scores, down to which questions fail. Round three is the table.

One result deserves its own line: the quant curve is not monotonic. On Qwen3-1.7B, Q4 scored 19, Q6 scored 16, Q8 scored 15. Citation discipline at this scale is a knife-edge behavior, and more precision does not buy it.

The fabrication hall of shame

The disqualifying failures were not dropped citations. They were invented ones. The gate’s off-corpus half exists to catch exactly this, and four models walked straight into it.

A concierge that invents a citation for a question it was never supposed to answer is worse than no concierge. These were rejected on that alone — the covered scores stopped mattering.

The one that was one question short

Qwen3.5-2B closed the sweep at 20/21, half the latency and half the memory of the incumbent. Its single miss was an honest one: a refusal on the compound RabbitMQ-or-Kafka question, the exact class of multi-source answers where small models drop their [n] markers.

By then the pattern was clear. Everything at or under 1.2 GB failed the contract — by dropping citations, by ignoring the format, or by inventing them. The discipline does not come from size, from precision, or from being a citation specialist (Pleias is one, and scored zero). So the best base got a teacher.

Teaching the contract

The dataset came from the incumbent 4B acting as teacher over the real retrieval pipeline. It invented 174 visitor questions in three languages — 77 simple, 65 off-corpus, 32 compound — and answered them under the exact production prompts. Every completion passed through the production citation policy before entering the dataset; anything that would not survive the gate never became a lesson. The 21 golden questions were excluded from the dataset, so the final score measures how the model generalizes to questions it never trained on.

The first recipe failed in a useful way. Three epochs with loss over the whole sequence scored 15/21 — worse than the base. The model had learned the shape of a cited answer so well that it started refusing answers it used to give.

The second recipe changed two things: loss over the completion only, and gentler training — two epochs, learning rate 7e-5, about five minutes on one consumer GPU. Result: 21/21 covered, 3/3 honest refusals, deterministic across re-runs, at ~310-355 ms and 1.2 GB.

Production, verified

The swap was one model file and one environment variable to pin the chat template to non-thinking mode. Live verification: the compound question the base refused now answers covered, citing both the CV and the Kafka repo. The eval gate runs the full battery inside the container on every deploy, and rollback is one line — the old 4B still sits in the models folder, just in case the gate and I ever disagree again.

Final scorecard: 21/21 covered, 3/3 honest refusals, ~310-355 ms p50, 1.2 GB resident — half the latency and half the memory of the model it replaced. The model, the dataset and the three training scripts are public.

Receipts

The fine-tuned model is on Hugging Face, with the dataset and the pipeline. The training and merge scripts live in the ask-api/finetune directory of the jordimp repo, next to the eval gate they had to pass. And the concierge is downstairs, by the door — ring the bell and ask it anything. Now you know how its brain was chosen.