Every benchmark post on this site rests on the same two machines, the same harness and the same set of rules. Rather than repeat all of it in each article, it lives here. Articles link back to this page; this page is kept current as the setup changes.
If you only read one section, read the rules. Every one of them exists because a measurement I had already published turned out to be measuring something other than what I claimed.
The two machines
Both runs use the same GPU model. Almost nothing else is the same — and that is the point. If two boxes this different produce the same bytes, the result is not an artefact of one particular machine.
| Workstation | Headless container | |
|---|---|---|
| Role | desktop, monitor attached | LXC on a Proxmox node |
| GPU | RTX 3090, 24 GB | RTX 3090, 24 GB |
| NVIDIA driver | 610.88 | 580.126.09 |
| CPU | i7-7820X, 8C/16T | i7-1165G7, 6 threads visible |
| RAM | 128 GB | 24 GB |
| OS | Windows 11 Pro | Ubuntu 24.04 LTS |
| Model store | SATA SSD | local disk, not NFS |
| Ollama | 0.32.5 | 0.32.5 |
One difference matters more than it looks: the attached monitor costs roughly 1.8 GB of VRAM before a single model is loaded. For any model that does not comfortably fit in 24 GB, the workstation therefore spills more onto the CPU than the headless box does. That is not a footnote — it is the single most common reason two otherwise identical runs disagree.
What gets measured
- Throughput — generated tokens per second, from Ollama’s own
eval_countandeval_duration. Reported as a median, never a mean. - Hit rate — a script checks whether the answer is right. Not whether it reads well.
- Format compliance — did the answer arrive in the shape that was asked for? Tracked separately and never folded into the hit rate.
- Cost — tokens produced and wall-clock seconds. A technique that adds nothing and costs nothing is a different result from one that adds nothing and costs half again as much.
- VRAM and CPU offload — from Ollama’s
/api/ps, recorded per cell.
What deliberately does not get measured
Quality of prose. Helpfulness. Tone. There is no score here for “the marketing copy is good”, because I cannot measure that — and neither can a second language model asked to grade the first one. LLM-as-judge introduces a second model’s variance and dresses it up as a number. Everything scored on this site is something a script can check against a fixed answer: does the JSON parse, does the anchor resolve to an ID that exists, is the total exactly 985.77, does every claim carry a source marker.
The price of that discipline is real: these benchmarks reward instruction-following and cannot tell you which model is more pleasant to work with. I would rather report a narrow thing accurately than a broad thing vaguely.
The rules — each one from a mistake
Reserved context is not filled context
Setting num_ctx to 128k and then sending a 30-token prompt measures the reservation of the KV cache, not its use. I published “context is nearly free” on the strength of exactly that mistake and had to take it back. Every context measurement now logs the actual fill level from prompt_eval_count and states it in the result.
Warm up after every context change
Ollama reloads the model whenever num_ctx changes, so the first measured condition carries the loading cost. In one smoke test that made an empty context look slower than a full one — 85.7 against 139.2 tok/s. One discarded request after every change, then measure.
The prompt cache invalidates prefill numbers
Repeat a prompt and the cache serves it: I once recorded a prefill rate of 295,631 tok/s, which is skipped work, not speed. Filler text is varied per repetition. Where prompts unavoidably share a prefix, prefill is simply not reported. Generation throughput stays valid either way.
Fix the output length, or the comparison is not paired
With a free-running answer length, different numbers of requests fall through a filter on each host, and the two sides stop being comparable — one run was ruined by exactly this, 36 usable cells against 24. num_predict is fixed, and the prompt token counts of both hosts are checked against each other before any difference is reported. If they do not match, there is no comparison to report.
Numbers come from the raw records
Every request is written to a JSONL file as it completes. All published figures are recomputed from those files, never read off a run log or a summary. Summaries are for looking at while a run is in progress.
Read the outputs, not just the scores
A scoring routine is a piece of software and can be wrong. Two examples from this year: a check demanded bullet points the task had never asked for, dropping a correctly answering model from 100 % to 69 %. And a check looked for a result line but did not recognise it when the model wrapped it in a Markdown heading — models that had solved the task were recorded at zero. Both were found by reading the actual output. Every scoring routine is now read against real answers before it is allowed to grade a run, and every output is kept in full so that anyone can contradict the score.
Truncation is not a bad answer
A model that runs into the token ceiling has not answered badly, it has not finished. Those cells are counted and flagged separately. A model that hits the ceiling in every single run gets its own note rather than a quiet place at the bottom of the table.
CPU offload breaks reproducibility
As long as a model fits entirely in VRAM, a fixed seed at temperature 0 reproduces across machines. Once part of it runs on the CPU, that stops being true — and how much runs on the CPU depends on what else is using the card, including the desktop. Any model that does not fit is marked as such and excluded from cross-machine claims.
Line endings are not a finding
The newest rule, and a cheap lesson. Comparing outputs between the Windows host and the Linux host reported that 97 of 99 files differed — a spectacular result that was entirely my own file writing: CRLF on one side, LF on the other. Normalised, the real number was 90 of 99 identical. Harnesses now write \n explicitly. When a measurement is dramatic, suspect the measurement first.
How to read the numbers
- Scores are fully solved cells where possible, not averaged partial credit. A half-correct invoice total is a wrong invoice total.
- Each run is three seeds (42, 43, 44) at temperature 0, so a single lucky answer cannot carry a model.
- Medians, not means, for anything timing-related.
- Everything is measured with
thinkdisabled. For reasoning models that is a statement about the model with its thinking mode off, not about the model. - Where a model has fewer cells than the others — a time budget ran out, a run was cut short — the table says so.
Corrections
Where a later article contradicts an earlier one, the later one wins, and both say so. Corrections are made in the original article rather than quietly in a new one. Two published claims have already been withdrawn this way; both are named in the rules above rather than hidden.
If you think a number here is wrong, the raw outputs exist. Ask for them.