Until today, every measurement in this series ran on a single graphics card. 24 gigabytes, and the boundary was clear: anything larger simply went unmeasured. As of today there are two RTX 3090s in the same machine. 48 gigabytes. Which makes a few things measurable that could previously only be asserted.
The result in one sentence: the smaller of the two models solves all 48 cells completely correctly and is nine times faster than the model twice its size — and that result is exactly what makes the test set useless for the next round.
The four tasks
Every task has exactly one correct answer — not a good-looking one. That is the condition under which machine scoring is possible at all. And every task demands a final line in a fixed shape, so the answer can be read without interpretation. The task names below are the identifiers used in the data files.
rechnung— multi-stage price calculation. Three line items with different quantities and prices. On top of that, in this order: per-item discounts above certain quantities, then 3 per cent early-payment discount on the subtotal, then a shipping fee that is waived above a net goods value, then 19 per cent VAT. The question is the final gross amount to two decimal places. Exactly one amount is correct.plan— small assignment problem. Four talks into four time slots, one talk per slot. Four constraints of the form “A is earlier than B”, “C is neither at 9 nor at 12”, “D is immediately after C”. Exactly one assignment satisfies all four. All four slots are scored individually.extrakt— extraction into JSON. Six fields must be pulled from an informal note: invoice number, date, full company name, gross amount, currency code and due date. The last one is not in the note at all — it has to be computed: invoice date plus payment terms, across a month boundary. Six field-exact checkable values.raetsel— constraint puzzle. Five colleagues in five seats, each with exactly one project and exactly one drink. Eleven constraints, including adjacency (“immediately to the right of”). All fifteen assignments are asked for — name, project and drink per seat. Fifteen individually checkable values.
“Fully correct” in the tables below means: every checkable value of a task is right. For rechnung that is one value; for raetsel it is fifteen. An answer with fourteen of fifteen correct assignments does not count as solved.
The four variants
Each task is run against four versions of the same prompt. Task text and format block are word for word identical in all four. The only difference is what is added before or after:
| Variant | What is added to the prompt |
|---|---|
nackt (“bare”) | Nothing. Only the task and the format block. This is the baseline. |
schritte (“steps”) | One sentence before the format block: “Think step by step.” |
rolle (“role”) | A role sentence before the task, matched to it — for rechnung, for instance: “You are an experienced accountant and have been checking quotations without error for twenty years.” |
beispiel (“example”) | A fully worked analogous task with its solution, placed between task and format block. So a solved case to imitate, not the solution to the actual task. |
This makes the prompts differ in length — beispiel is consistently the longest. That is intended and is measured, because an addition costs input length before it does anything useful.
What was measured
Two models that cannot run on a single card:
llama3.3:70b— 39.6 GB, dense 70-billion-parameter model, quantisation q4_K_Mqwen3.5:35b-a3b-q8_0— 36 GB, mixture-of-experts with 3 billion active parameters, quantisation Q8
Four tasks times four variants times three seeds gives 48 cells per model. num_ctx 32768, num_predict 4000, temperature 0, thinking mode off. One warm-up run per model, its result discarded. The setup is the one from runs v1 through v3, unchanged — otherwise no figure would be comparable with an earlier one.
No cell was truncated. All 96 are present.
The numbers
| Model | Cells | fully correct | mean share | Speed |
|---|---|---|---|---|
llama3.3:70b | 48 | 33 | 0.931 | 10.96 tok/s |
qwen3.5:35b-a3b-q8_0 | 48 | 48 | 1.000 | 96.96 tok/s |
“Mean share” is the average of correctly solved sub-values across all cells — 0.931 therefore means that on average 93.1 per cent of the checkable individual values are right, even though only 33 of 48 answers are fully correct.
In wall-clock time per task, averaged across all variants and seeds:
| Task | llama3.3:70b | qwen3.5:35b-a3b |
|---|---|---|
extrakt (six fields from a note) | 32.9 s | 3.0 s |
plan (four talks into four slots) | 42.8 s | 8.6 s |
rechnung (discount, terms, shipping, VAT) | 57.1 s | 10.1 s |
raetsel (five people, fifteen values) | 166.0 s | 23.6 s |
On the constraint puzzle that is 166 seconds for a mostly wrong answer against 23.6 seconds for a correct one.
Where the 70B fails
The 15 failures of llama3.3:70b are not spread out. They sit in two nests. The table shows how many of the three runs per field were fully correct.
| Task | nackt(task only) | schritte(+ “think step by step”) | rolle(+ role sentence) | beispiel(+ solved analogue) |
|---|---|---|---|---|
plan | 3/3 | 3/3 | 3/3 | 3/3 |
rechnung | 2/3 | 3/3 | 3/3 | 3/3 |
extrakt | 3/3 | 0/3 | 3/3 | 3/3 |
raetsel | 1/3 | 0/3 | 0/3 | 0/3 |
The constraint puzzle breaks the model in all four variants — 1 out of 12. And the extraction task, which it solves flawlessly in three variants, fails completely the moment “think step by step” is prepended. The rest of the matrix is spotless.
What fits into 48 gigabytes — and what doesn’t
This is the practically most useful part. Two 3090s sound like a lot. The current generation of large models is out of reach regardless:
| Model | Size | fits in 48 GB? |
|---|---|---|
qwen3.5:27b-q8_0 | 30 GB | yes, with room |
qwen3.5:35b-a3b-q8_0 | 39 GB | yes |
llama3.3:70b (q4_K_M) | 43 GB | barely — see below |
gpt-oss:120b | 65 GB | no |
mistral-medium-3.5:128b | 80 GB | no |
qwen3.5:122b-a10b | 81 GB | no |
nemotron-3-super:120b | 87 GB | no |
The threshold is not 48 but closer to 44 gigabytes. The rest goes to the context cache and the display output.
The 5.1 per cent nobody sees
For each loaded model, Ollama reports how much of it actually sits in graphics memory. The difference is the part running on the CPU:
| Model | total | in VRAM | offloaded |
|---|---|---|---|
llama3.3:70b | 48.95 GB | 46.45 GB | 5.1 % |
qwen3.5:35b-a3b-q8_0 | 38.48 GB | 38.48 GB | 0.0 % |
Nothing in Ollama’s own output hints at this. No notice, no warning. The model loads, it answers, and one twentieth of it is being computed by the CPU. Anyone who does not query that figure will read 10.96 tok/s as the performance of two RTX 3090s.
Half of this is my own setup. num_ctx was pinned at 32768, because otherwise no figure in this series would be comparable with an earlier one. That very context cache is what pushes llama3.3:70b over the edge. With a smaller context it would fit entirely. So the 10.96 tok/s are not the best this model can do on this hardware — they are the best under a constraint I imposed for the sake of comparability. How much it would be without that constraint, I do not know; it was not measured.
Two older findings, independently confirmed
The seed does nothing, the position of the run does
Two days ago this blog reported that seeds have no effect on these models, and that the first run is what differs instead. That was measured across eleven models on a single graphics card. Here is the same signature, on new hardware and on a model that did not appear in that dataset at all — on the constraint puzzle, where fifteen values are checked individually:
| Variant | Seed | Points | generated tokens |
|---|---|---|---|
nackt | 42 | 15/15 | 1404 |
nackt | 43 | 10/15 | 1508 |
nackt | 44 | 10/15 | 1508 |
schritte | 42 | 13/15 | 2122 |
schritte | 43 | 13/15 | 1725 |
schritte | 44 | 13/15 | 1725 |
Seed 43 and seed 44 produce identical results — same score, same token count, in both variants. Seed 42 differs. And seed 42 is the first run in each case. If the seeds were doing anything, all three would differ. They do not.
“Think step by step” hurts extraction
On Monday this blog reported that the phrase hurts on one task in four — the extraction task, that is, pulling six fields out of an informal note. The obvious objection was: the models measured were too small, a larger one would absorb the instruction. llama3.3:70b is more than twice the size of anything measured back then. Without the phrase, three of three fully correct. With it, zero of three. On the same task.
The objection is settled. Not refuted for all model sizes — but for this particular increase in size, which is the one that was actually in question.
What this does not establish
The obvious headline would be “mixture-of-experts beats dense”. It is not there, because the data do not support it. The two models differ in at least four ways at once:
- Architecture — MoE with 3 billion active parameters versus dense
- Quantisation — Q8 versus q4_K_M
- Release date — eighteen months apart
- Execution — one fits entirely into graphics memory, the other does not
Which of those four differences produces the gap cannot be determined from two models. What is established is only this: on this task set, qwen3.5:35b-a3b at Q8 is flawless and roughly nine times faster than llama3.3:70b at q4_K_M.
The test set has hit its ceiling
48 out of 48 is not a good result. It is a measurement problem. About a model that solves every cell exactly one statement can be made: it did not fail. How much headroom it has, whether another model would be better, where its limit lies — none of that is measurable any more. A task set the subject solves completely only measures itself.
So since this afternoon a second set of six tasks has been running. The scoring rules were fixed in writing before the first cell was measured. What the subject is asked to do in each:
rechnung2— the same kind of invoice as above, but with two VAT rates side by side (19 and 7 per cent), discount before early-payment terms, and rounding only at the very end, per VAT rate. Three amounts are asked for: gross per rate, and the total.plan2— six talks into six time slots instead of four into four. Five constraints, including “F lies strictly between A and B”. Six checkable values.extrakt2— nine fields from a note with three deliberate traps: three dates, only one of which is the invoice date; a “represented by Mr …” who is not the customer; an IBAN with spaces that must be normalised. Plus two computed deadlines and one computed gross amount.widerspruch— distribute five parcels across two vans under constraints that look satisfiable but are not. The correct answer is “no solution” — together with the two total weights the constraints would require. Guessing the get-out without doing the arithmetic does not produce those figures.trace— given a small Python function with a loop, a Fibonacci sequence and a divisibility branch, name two of its return values. No room for interpretation in the scoring.raetsel— the constraint puzzle from above, unchanged. An anchor: this single task is the only thing linking the old and new datasets.
Before anything was measured, the scorer itself was tested: the known correct answers must score full marks, tolerable spellings likewise (1.924,70 EUR as readily as 1924.70), the built-in traps must not, and an answer with no result line must score zero. A scorer that fails to recognise the correct answer produces a systematic error across the entire run — and that does not stand out in the data, because everything there looks uniformly wrong.
Three mistakes of my own
First. In an interim summary, my analysis table showed an empty cell for the constraint puzzle, and I read it as “zero of eight correct”. That was a formatting artefact of the analysis, not a measurement. It was in fact one of eight. A zero that comes from an empty table cell is not a result — it is a missing value, and the two look identical.
Second. The 5.1 per cent of offloading above. I pinned the context cache at 32768 for the sake of comparability and did not consider that this very choice pushes one of the two models past the memory limit. The result is valid, but it is not a comparison of two models under equal conditions — one ran handicapped, and by my setup rather than by its own properties.
Third, and the most instructive: on the same day, a data backup was running whose script writes an overall verdict at the end. The verdict read “OK”. Two lines above it, in the same report, stood the return code 9 — which in that family of tools means “files could not be copied”. The check for exactly that case was present in the script and missed, because the result list contained not only the results but also the function’s status messages, and those have no field for a return code.
It was not the copy that failed but its assessment — and that is the worse case, because a failure gets reported as a success. It is the same thought that forces the self-test of the scoring functions above: a program reporting success is not evidence of success. The evidence is a second, independent look at the result. The backup script had none. The test harness now does.
The setup, for anyone re-running this
- Two RTX 3090, driver 610.88, 48 GB combined. No NVLink — the bridge is not fitted, both cards report all links inactive.
- The second card is connected with eight PCIe lanes instead of sixteen. Barely noticeable for inference, since little moves between the cards — very noticeable when loading a 40 GB model.
- Ollama 0.32.5,
OLLAMA_FLASH_ATTENTION=1,OLLAMA_KV_CACHE_TYPE=q8_0. - Under load the cards reached 77 and 73 degrees at 201 and 221 watts — out of a 350 watt specification. The upper card runs three to four degrees warmer throughout.
At the time of writing, the harder set was still running. Its numbers therefore appear in the next post rather than this one — complete or not at all. In part four of this series I wrote a section after two models out of four, and the third reversed the verdict. Once is enough.