Before any of the measurements in this series, the ground rules. Not because rules are interesting, but because a benchmark without them is a number with no way to check it — and I would rather you were able to tell me I am wrong.
Everything here is the standing setup. The How I Measure page carries the same thing in its always-current form, including anything I learn after writing this; where the two differ, the page is right.
Two machines, one GPU model in common
Every measurement runs twice, on two hosts that share a GPU model — though not the same board maker — and almost nothing else. One is a Windows workstation with a monitor attached, plenty of RAM and a desktop-class CPU. The other is a headless Linux container on a Proxmox node with a fraction of the memory and fewer cores. Same RTX 3090, an EVGA card on one side and a Zotac on the other, same Ollama version, different everything else.
That is deliberate. A result that only holds on one box is not a result, it is an anecdote about that box. If the two disagree, the disagreement is the finding and gets its own section rather than being averaged away.
One consequence is worth knowing in advance: the attached monitor occupies VRAM before a single model is loaded. For any model that does not comfortably fit in 24 GB, the workstation therefore pushes more of it onto the CPU than the headless box does. As the opening article put it, that cliff is the first thing to check before blaming anything else — and it is why the two hosts can legitimately produce different answers from the same seed.
What gets scored, and what does not
Only things a script can check against a fixed answer. Does the JSON parse. Does the anchor point at an ID that exists in the document. Is the total exactly the figure I worked out by hand beforehand. Does every claim carry the source marker the task demanded.
There is no score for whether the text reads well. I cannot measure that. Nor can a second language model asked to grade the first one — that approach imports a second model’s variance and presents it as a number, which is worse than not measuring at all, because it looks rigorous.
The price of that discipline is real and I would rather name it than hide it: these benchmarks reward following instructions. They cannot tell you which model is nicer to work with, and they will sometimes rank a pedantic model above a smarter one. A narrow claim that holds beats a broad one that does not.
The rules the numbers have to survive
- Fixed parameters, or the comparison is not paired. Context size and maximum answer length are pinned on both hosts. Before any difference between the two is reported, the prompt token counts are checked against each other. If those do not match, the two runs are not comparable and there is nothing to report.
- A discarded warm-up run before measuring. Changing the context size makes the runtime reload the model, so the first measured condition would otherwise carry the loading cost and look slow for a reason that has nothing to do with the condition.
- Prefill numbers are treated as suspect. Repeat a prompt and the cache serves it; the reported prefill rate then describes skipped work rather than speed. Where prompts unavoidably share an opening, prefill simply is not reported. Generation throughput stays valid either way.
- Every request is written to a raw record as it completes, and every published figure is recomputed from those records — never read off a progress log or a summary file.
- Three seeds at temperature 0, so that one lucky answer cannot carry a model. Medians rather than means for anything to do with timing.
- Hitting the token ceiling is not a wrong answer. A model that runs out of room has not answered badly, it has not finished. Those cases are counted and flagged separately instead of quietly sinking to the bottom of a table.
- Thinking modes are switched off. Where a model has one, the numbers describe it with that mode disabled. That is a real limitation and it will be repeated wherever it matters, not buried here.
Corrected on 2026-08-02: the three runs stand, the reasoning does not. Measured on both hosts, ornith:35b and qwen3.6:27b return byte-identical output across all three seeds — there the seed changes nothing at all. qwq:32b, gemma4:26b, qwen3:8b and deepseek-r1:14b, by contrast, produce different output from an identical seed. Three seeds therefore catch variance where variance exists, but they do not test the effect of the seed. The measurements are unaffected — only this justification is.
Addendum of 2026-08-03: “from an identical seed” is put too loosely. Re-measured, exactly one run per group differs — always the first, in 111 cases out of 111. Every later run is byte-identical to the others, across both machines as well (108 of 108 cell pairs). Five different seeds produce the same text as five repetitions of one seed: the seed has no effect, the position does. The measurements are unaffected.
Read the outputs, not just the scores
This is the rule I expect to earn me the most work. A scoring routine is a piece of software and can be wrong in a way that looks exactly like a model being bad. It can demand something the task never asked for. It can fail to recognise a correct answer because of how the model decorated it. Both failure modes produce a confident number and a wrong conclusion.
So: no scoring routine grades a run until I have read it against real answers, and every generated output is kept in full. Not a sample — all of them. If a score here is wrong, the evidence to prove it exists and you can ask me for it.
When I get it wrong
The opening article of this series made a promise: if a later article contradicts an earlier one, the later one wins and both say so. This is the mechanism behind that promise. Corrections go into the original article rather than quietly into a new one, and the reason for the correction gets named.
I would rather publish a smaller claim I can defend than a larger one I have to walk back. Experience suggests I will do both anyway.