A few hours ago I published a throughput table: nine local models, tokens per second, cleanly measured. Its closing paragraph said speed is one dimension of several and not the most important one, and that a quality measurement was still missing.
It is no longer missing. The throughput winner comes second-to-last on quality. The model that came last on throughput comes second.
The task
Each of the nine models got the same text: build a playable Tetris that runs in a browser and works on a phone. A single self-contained file, no network access at runtime, a fixed key map, a test hook for reading out the playfield. One call per model, no repair loop, no second chance.
The grading was done neither by the model nor by me, but by a hidden verifier that opens the artifact in a real browser and runs sixteen behavioural checks: does a piece fall on its own, does it move on an arrow key, does it rotate, do all seven shapes appear, does a full row disappear, does the game end. It measures pixels and real input events — never what the code claims about itself. The test hook cannot be faked: its return value is cross-checked against the rendered image. In calibration, a solution that merely pretends to have the hook therefore scores deliberately below an honest partial one.
The table
| Model | Throughput rank | Reasoning chain | Score | Checks |
|---|---|---|---|---|
| Qwen3.8 35B-A3B Distill (Q4_K_M) | 2 | 3,445 | 100.0 % | 16/16 |
| 27B rebuild (Q8_0) | 9 — last | 919 | 91.4 % | 15/16 |
| Qwen3.8 27B — in production | 7 | 55,187 | 85.7 % | 15/16 |
| Qwen3 Coder 30B | 4 | 0 | 62.9 % | 12/16 |
| Qwen3.6 35B-A3B MTP | 6 | 1,345 | 51.4 % | 10/16 |
| Nemotron 3.5 Lightning 30B-A3B | 1 — fastest | 82,812 | 34.3 % | 7/16 |
| Qwen3.8 35B-A3B Distill (IQ2_M) | 5 | 1,789 | 11.4 % | 3/16 |
| Qwen3.8 27B abliterated (Q6_K) | 8 | 104,673 | no artifact — budget spent inside the reasoning chain | |
| Ornith 1.5 35B-A3B | 3 | 113,152 | no artifact — budget spent inside the reasoning chain | |
The throughput ranks come from this morning’s measurement, run at the full context window; the reasoning-chain column is the character count of the thinking trace in this run. The bottom two rows are not empty because the models failed — they never got to the writing. What is behind that is further down; I went and measured it.
Exactly one model is near the top of both lists: the 35B-A3B distill in Q4_K_M, throughput rank 2 and the only perfect score. Everything else moves.
The biggest mover is the Q8_0 rebuild. In the throughput table it is the slowest model in the field, occupies the most memory and takes longest to load — I called it “the worst choice on every dimension except prefill” there. On that table’s dimensions, that is true. As an overall verdict it was misleading: here it comes second.
Moving the other way is Nemotron. 145.3 tokens per second in the morning grid — two and a half to three and a half times the production model’s throughput, depending on prompt length — and at a smaller VRAM footprint, 27.65 against 32.39 GiB. Quality: 34.3 %, sixth of seven scorable artifacts.
Five of seven build a Tetris that never clears a line
The more interesting number is not in the ranking but underneath it. The most common failures across all seven scorable artifacts:
| Check | fails |
|---|---|
touch_controls — playable on a phone |
5 of 7 |
line_clear — a full row disappears |
5 of 7 |
game_over |
3 of 7 |
gravity — piece falls on its own |
3 of 7 |
seven_pieces — all seven shapes |
3 of 7 |
Clearing full rows is Tetris. Without it you have a stacking game that loses after a minute. Five of seven artifacts do not have it — including the model in third place, which otherwise passes fifteen of sixteen checks. An artifact can get almost everything right and still not be the game that was asked for.
Phone controls are a different case: they were explicitly in the brief, and five of seven simply skipped them. Not a comprehension problem, an attention problem.
Filed late: with the reasoning chain off, both of them write
Two models spent their entire token budget inside the thinking trace and never started writing — exactly 32,768 tokens each, cut off, zero characters of answer. That does not read as “cannot build Tetris”. The other explanation — “never gets to the writing” — calls for the same run with the reasoning chain switched off. I went back and did it.
First, a control on whether the switch does anything at all: the same arithmetic question once with and once without the chain. With: 152 and 106 characters of trace, both answers correct. Without: zero characters of trace — and both answers wrong. So the switch works, and it costs something.
| Model | with reasoning chain | without |
|---|---|---|
| Qwen3.8 27B abliterated (Q6_K) | no artifact | 68.6 % · 13/16 |
| Ornith 1.5 35B-A3B | no artifact | 57.1 % · 11/16 |
Both build a Tetris. Both would have beaten the throughput winner, which lands at 34.3 % with its chain free. And both pass the phone controls that five of seven in the main run fail.
These two numbers still do not belong in the same column as the seven above: they come from a different condition. The control showed that switching off thinking costs these models correctness — so what stands here is “an artifact does come out”, not “this is how good they are”. What the measurement carries is this: the empty cell was an abort, not an inability.
For completeness: these two artifacts do not clear lines either. That makes seven of nine that fail at the mechanic which defines Tetris.
Reasoning-chain length predicts nothing — until it eats everything
The obvious guess would be that more thinking yields better code. Or the opposite. The measurement supports neither. Across the seven scorable artifacts the relationship between trace length and score is essentially absent — the rank correlation sits at −0.14, which is noise:
- Qwen3 Coder does no thinking at all (0 characters) and lands at 62.9 %.
- The winner at 100 % thinks 3,445 characters.
- The production model thinks 55,187 characters — sixteen times the winner — and still comes third.
- Nemotron thinks 82,812 characters and lands at 34.3 %.
So the length of a thinking trace tells you nothing about the quality of the result. What it does reveal is a failure mode of its own: a chain that does not stop consumes the budget the answer needed. Two models did exactly that, to a hundred percent.
My first explanation was that the slow models run into the time limit. It died immediately: in the same grid Ornith, at 126.6 tokens per second, is a good four times faster than the other failed model at 31.0 — and burns the budget just as completely. It is not about speed. These models do not stop.
Three failed attempts, all mine
Before a single number existed, the setup failed three times — and every time it was me, not the models. The harness booked all three as tooling failures rather than model failures. That distinction is exactly what it is for.
First: the coding agent. The obvious route was to let the models work through a coding agent, as the harness intends. All nine failed identically after 57 seconds with Request timed out. That is not the task’s time limit — that one is thirty minutes — but the agent’s own built-in request timeout, and it cannot be configured. The second available agent carries a model list that contains none of the nine. For a model comparison an agent is a confounder anyway; I dropped it.
Second: the token budget. At 16,384 tokens the first model was cut off — it had produced 33,080 characters of reasoning. For comparison: the same model came to about 1,200 characters in the throughput measurement the same day. The reasoning chain scales with the difficulty of the task, not with the model. A budget that suffices for a two-sentence summary is nowhere near enough for an implementation.
Third, and this is the genuinely interesting one: the task text is written for an agent. It says: “You are working in the current directory. It is empty. The files you leave on disk are the entire submission.” Handed to a chat model with no filesystem, it answered entirely logically:
“Looking at this task, I need to build a complete, self-contained Tetris game … Let me start by writing the game, then test it thoroughly.”
And stopped. 8,040 characters of reasoning, 221 characters of answer, finished normally. It announced what it would do with tools it did not have.
A task statement carries unspoken assumptions about the environment it runs in. The same text measures something different with an agent than without one — and without adaptation it measures nothing at all. So I changed only the delivery: emit the complete file in one code block, you have no filesystem. The brief with its four mandatory points stayed word for word. The verifier does not care how the file came to exist; it opens it in a browser.
What this measurement does not say
It is one run per model, at a temperature that admits chance. The gap between 100 % and 91.4 % may be noise; the order of the top places is not established. What it does carry is the coarse shape — that a model at 11.4 % and one at 100 % are not separated by luck, and that the throughput ranking does not predict the quality ranking.
The two late-filed numbers from the run without the reasoning chain are weaker still: a different condition, and likewise a single run. They answer a yes-or-no question — does an artifact come out? — and not a question of rank.
It also measures a single shot. In an earlier measurement with the same kind of task, a repair loop — feed the errors back, try again — lifted two cells from four to eight out of eight. Anyone using these models day to day is using a loop. What is here is the first answer, not the best one.
What I take from it
This morning’s throughput table was correctly measured and still worthless as a basis for deciding. Not because the numbers are wrong, but because they answer a question nobody has. Nobody wants to know how fast a model produces wrong answers.
Switching the production model is still not due — but now for a better reason. It is not the fastest, but it sits third, and its single failed check is the same one five of seven fail. The fastest model in the house would have been a step backwards, and the throughput table on its own would have recommended it.
The seven artifacts from the main run are public and playable. Arrow keys. The one at 11.4 % is worth opening once — at first glance it looks like Tetris.