Ollama 0.32.5 was out, my workstation was still on 0.31.2, and I had just finished benchmarking 26 models on it. So I had a rare opportunity: the same machine, the same models, the same settings, and a version number as the only variable.
The result is a null result, which is exactly why it seems worth writing down. Eleven of twelve models moved by less than 3 %. The twelfth broke.
Holding everything else still
Same RTX 3090 on native PCIe, driver 610.88, Windows 11, KV cache at q8_0, flash attention on. Twelve models spanning nine architectures from 4.9 GB to 19.7 GB, mixture-of-experts and dense, thinking models and instruct models, plus one vision model.
Eleven tasks per model, each run three times with different seeds. That last part matters more than it sounds: at temperature 0 the generation is near-deterministic, so repeating with the same seed buys you almost nothing. Different seeds tell you whether a model solves a task reliably or got lucky once.
One setting needed pinning explicitly. The two machines I eventually want to compare disagreed on the default context length — 65,536 on one, model-default on the other. A larger context means a larger KV cache and lower throughput, so leaving it alone would have reported a configuration setting as a platform difference. Anything that affects the measurement goes in the request, not the environment.
The numbers
Throughput is the median over generations of 50 tokens or more. Scores are solved-out-of-asked across three seeds.
| Model | Reasoning | Thinking | Code | 0.31.2 | 0.32.5 | Δ |
|---|---|---|---|---|---|---|
qwen3-coder-abliterated | 15/15 | rejected | 9/9 | 144.7 | 149.4 | +3.2 % |
qwen3-vl-abliterated | 15/15 | 15/15 | 9/9 | 138.0 | 141.4 | +2.5 % |
laguna-xs-2.1 | 8/15 | 11/15 | 9/9 | 145.0 | 141.0 | broken |
glm-4.7-flash | 6/15 | 12/15 | 9/9 | 124.6 | 124.9 | +0.2 % |
ornith:35b | 15/15 | 15/15 | 9/9 | 117.9 | 119.5 | +1.4 % |
qwen3:8b | 12/15 | 15/15 | 9/9 | 119.1 | 116.1 | −2.5 % |
gemma4:26b | 15/15 | 15/15 | 9/9 | 103.2 | 102.7 | −0.5 % |
deepseek-r1:14b | 12/15 | 15/15 | 3/9 | 69.2 | 68.2 | −1.4 % |
devstral:24b | 15/15 | rejected | 9/9 | 49.7 | 48.4 | −2.6 % |
qwen3.6:27b | 15/15 | 15/15 | 9/9 | 39.1 | 39.2 | +0.3 % |
qwq:32b | 8/15 | 15/15 | 0/9 | 19.7 | 35.2 | context |
Everything sits between −2.6 % and +3.2 %. The one large number, qwq:32b at +78.7 %, is not the version at all — the baseline ran at 64k context and the comparison at 8k, and qwq is the only model in the set that offloads to CPU at large context. Every other model showed zero offload, which is precisely why nothing else moved.
The other thing this table shows: eleven of twelve models reproduced their scores exactly against a single-sample run from the previous night, including partial scores like 3/9 and 12/15. That is the strongest evidence I have that the task set measures a property of the models rather than of the sampling.
The one that broke
laguna-xs-2.1 hit the token cap on every single reasoning request — 30 out of 30, at exactly 900 and exactly 3,000 tokens. On 0.31.2 the same model averaged 215 and 897 tokens and never truncated once.
The JSON task shows what is happening:
{"name": "Tobias", "city": "Berlin", "age": 42}</assistant>{"name": "Tobias", "city": "Berlin", "age": 42}</assistant>{"name": ...The model produces the correct answer, emits its end marker, and then keeps going — regenerating the same answer until the budget runs out. The stop token is not being honoured.
The likely culprit is release 0.32.3, whose notes say it “added chat, thinking, and tool calling support for Laguna 2.1 models”. The release that was supposed to wire this model up properly appears to have got its stop tokens wrong. I confirmed it is not a context effect by running the same model at 8k and 64k — identical breakage at both.
Its 8/15 and 11/15 scores in the table are meaningless. They come from my “last number in the reply” rule landing somewhere in repeated text. The honest label is broken, not worse.
“Disable thinking” is a request, not a guarantee
This was the most useful thing the exercise turned up, and it has nothing to do with versions. Ask a model to skip its chain of thought and one of three things happens:
- Honoured — token count drops by a factor of 27 to 812. One model went from 2,435 tokens to 2.
- Silently ignored — token counts identical in both modes. The flag is accepted and does nothing.
- Rejected — HTTP 400. The model has no thinking mode and says so.
The middle case is the dangerous one, because everything looks like it worked. I initially blamed the Ollama version for it. It is not the version — the same model ignores the flag on 0.31.2 and 0.32.5 alike. It is a per-model property, in at least one family caused by a template that ships without the thinking-control logic.
The test costs two requests: send the same prompt with thinking on and off, compare the token counts. If they are within a few percent, the flag did nothing and any direct-versus-thinking comparison you draw is measuring an axis that model does not have.
There was one genuine improvement between versions, and it is of this kind: qwen2.5vl:7b now rejects the flag with a 400 where it used to accept and discard it. An explicit error beats a silent no-op every time.
A tag name is not an identity
While preparing the second half of this comparison I pulled a model that another machine already had, expecting a clean like-for-like test. The two copies of the same tag turned out to be different files — 17.28 GB versus 18.25 GB, and different model families. The upstream tag had been re-pointed at a different base model between the two pulls.
It got stranger. On the other machine, three tags — one named “coder”, one named “coder-abliterated”, one named “vl-abliterated” — all pointed at a single blob, and that blob was a vision-language model, not a coder at all. A throughput comparison I had already drawn between the two machines was therefore comparing a vision model against a coding model and calling the difference “hardware”.
Before comparing anything across machines: check the manifest digest, then the actual weight blob, then the model family. The tag proves nothing.
Two mistakes of my own
Halfway through I improved the throughput metric — from a mean across all requests to a median across generations of 50 tokens or more, because on a two-token answer you are measuring request overhead rather than decode speed. Good change. Then I compared the new figures against the old summary values.
Short answers have low throughput, so the old mean was depressed, so every model appeared to gain 3–11 % from the upgrade. I reported that for four models in a row before the uniformity bothered me — four models from four different vendors all improving by a similar amount is not how software works. Recomputed from the raw per-request data, the gains vanished entirely.
The rule I should have followed: never compare a number produced under one metric against one produced under another. If the metric changes, recompute everything it will be compared against — which is only possible if you kept the raw data.
The second mistake was subtler. Recomputing, I dropped one filter and accidentally pulled the overnight soak-test requests into the baseline medians — 66 samples instead of 10 — which flipped signs on several models. Caught it because one value disagreed with an earlier calculation by more than it should have.
So should you upgrade?
Yes — but not for speed. There is no speed here. Upgrade for bug fixes and for support of models you actually want to run.
And then check that nothing broke on the way, because something did. A model that silently never stops generating is not going to announce itself in a changelog. It shows up as a job that used to finish and now runs to the token limit, or as a bill that got larger for no visible reason.
Next up: the same twelve models on a second RTX 3090 — same card, but attached over Thunderbolt to a Linux host with a quarter of the RAM and a mobile CPU. Same GPU, entirely different stack. And a proper context sweep across 8k, 32k, 64k and 128k, because the one large number in this post is still a context effect I have argued rather than measured.