One model sits at the bottom of this series’ prompt benchmark with 35 percent. The number was measured honestly. The conclusion you would draw from it would still have been wrong — and wrong in both directions at once.
The model is laguna-xs-2.1, 33.4 billion parameters at 4-bit. In the main prompt comparison I reported its column separately instead of averaging it in, on the grounds that it hits the token ceiling in nearly every cell, so what gets measured there is the ceiling and not the prompt. That reasoning was correct. It just wasn’t the end of the story — it was the beginning.
Not nearly every cell. Every cell.
48 of 48 cells at the ceiling, on both machines. And not roughly the same on each — identical: 117 of 336 points, 15 of 48 format-compliant, 88, 88, 0 and 30 percent across the four tasks. The Windows workstation and the Linux container produced the same digits. Two computers that share almost nothing except a GPU model — not even the board maker, an EVGA on one side and a Zotac on the other. They arrived independently at the same result. That is not one machine’s quirk. That is a property.
So I did what the How I measure page states as a rule: read the outputs, not just the scores. What is in them leaves no room for interpretation.
ERGEBNIS: {"rechnungsnummer": "RG-2026-0417", ... "faellig_am": "2026-03-13"}</assistant>ERGEBNIS: {"rechnungsnummer": "RG-2026-0417", ...The model solves the task. Completely, including the due date computed correctly across a month boundary. Then it emits </assistant> — its own marker for “I am done” — and starts over. And again. And again, until the 4000-token limit cuts it off. That marker appears 684 times across the 48 outputs. It is the only marker of its kind; there is no second candidate to puzzle over.
The cause is embarrassingly mundane. This model’s Ollama packaging contains no stop token at all. Its template is a single line, {{ .Prompt }}, with no role markers. The model says “done” and nobody is listening. It was not the thinking mode — that was switched off throughout, and the field for reasoning steps stayed empty in every run.
Corrected on 2026-08-02: the think switch does not reliably control whether the model reasons — for qwq:32b and deepseek-r1:14b it demonstrably has no effect. “Switched off” therefore does not apply to every model here. Details on How I Measure.
Two lines
FROM laguna-xs-2.1:latest
PARAMETER stop "</assistant>"That is the whole fix. The derived model shares its weights with the original, so it costs no extra disk, and it leaves the original untouched — which matters, because the original is the evidence and has to stay reproducible. Asked a plain “Hallo”, it previously answered with 2000 tokens and 80 repetitions. Afterwards, with ten tokens: Hello! How can I assist you today? Done.
What the fix changes
| original | with stop token | |
|---|---|---|
| cells at the token ceiling | 48 of 48 | 3 of 48 |
| format compliance | 31 % | 83 % |
| points | 117 / 336 | 215 / 336 |
| tokens generated | 192,000 | 45,775 |
| compute time | 1594 s | 326 s |
A quarter of the tokens, a fifth of the time, nearly double the score. So much for the half of the story you would expect.
The half you would not expect
The defect did not only push scores down. It pushed one up, in the same measurement, on a different task.
| task | points before | points after | format |
|---|---|---|---|
| rechnung (multi-step arithmetic) | 88 % | 75 % ↓ | 25 % → 100 % |
| plan (assignment problem) | 88 % | 88 % | 0 % → 100 % |
| extrakt (six fields to JSON) | 0 % | 100 % ↑ | 100 % → 100 % |
| raetsel (logic puzzle) | 30 % | 39 % | 0 % → 33 % |
On rechnung the model was better before. The reason is in my own scoring code: it takes the last ERGEBNIS marker in the text. For a single answer that is sensible — models sometimes correct themselves, and the final version is the one they mean. For eighty repetitions it is something else entirely: the model effectively gets eighty attempts and the last one is graded. That is no longer a measurement. It is a raffle with a lot of tickets.
On extrakt the same rule swung the other way. There the final repetition was cut off mid-JSON by the token ceiling. Unparseable, therefore zero — for a model that in truth solves that task completely in every single cell. From 0 to 100 percent, with nothing about the model having changed.
That yields a rule which now lives on the methodology page: “the last marker counts” assumes exactly one answer. Where a model runs on, the scorer has to say so and void the cell, rather than grade one repetition out of many. I had detected the runaway generation and flagged it — what I had not seen was that it pulls the floor out from under my own scoring rule.
What this does not claim
- Three cells still hit the ceiling — all three on the logic puzzle with the step-by-step variant, across all seeds. That is no longer a packaging defect but a model talking itself into a corner on the hardest task: 2202 tokens at the median, against 97 on the easiest. It stays in the table and does not get explained away.
- The available update does not fix this. Comparing the manifests: the template layer is identical locally and remotely, while the weights differ. So the update swaps the model, not the packaging. I have not tested the new weights — I am only saying there is no reason to expect them to solve this problem, and that swapping a model mid-series destroys the comparability of everything measured so far.
- This says nothing about whether laguna is a good model. It is a statement about four tasks, with thinking mode off, in one particular packaging. 64 percent is the more honest number than 35 — that is still a long way from being the true one.
Why this is here
Because the comfortable route was wide open. The column was already reported separately, with a factually accurate justification. Nobody would have asked. A model sitting at 35 percent at the bottom of a table is a completely unremarkable row.
The uncomfortable part is not the model’s fault. It is that my own scoring rule collapsed underneath it without anything looking like an error. The number was plausible, the justification was correct, the log was clean. I only found it because I read the generated outputs — the same rule that already stopped this series from scoring a correctly answering machine at zero percent.
To be sure, I repeated the corrected measurement on the second machine. Again 215 of 336 points, again 40 of 48 format-compliant, again the same three cells at the ceiling — and exactly the same 45,775 generated tokens. Not approximately the same: all 48 outputs are character-for-character identical between the two computers. Different CPUs, different operating systems, same GPU model on boards from two different makers. The only difference was the wall clock, 326 against 338 seconds — precisely the thing that is allowed to differ between two machines.
The same holds for the broken state: there, too, both hosts independently reached the same 117 of 336. What was measured here is, in both cases, not one box’s quirk. It is the model.
I would rather make a smaller claim I can defend than a larger one I have to withdraw. Here it was both at once: a number that was too small, and inside it a second one that was too large.