Yesterday this blog reported that the test set had hit its ceiling: one model solved all 48 cells, and about a model that solves everything nothing can be said except that it did not fail. So: a harder set. Six tasks, three models, 108 cells, 97 minutes. The ceiling is gone — and what appears underneath is less comfortable than a leaderboard.
The result in one sentence: given a task that has no solution, the largest of the three models invents one in five runs out of six — and makes 14 kilograms of freight disappear to do it.
The six tasks
Each has exactly one correct answer and demands a final line in a fixed shape. The names are the identifiers used in the measurement data.
rechnung2— invoice with two VAT rates. Three line items: 14 hours of consulting at EUR 95.00 net (19 %), three technical books at EUR 32.00 net (7 %), one maintenance fee of EUR 480.00 net (19 %). From 10 hours, a 12 per cent discount on the consulting; then 2 per cent early-payment discount on the total; rounded only at the very end, per VAT rate. Asked for: gross amount per rate, and the total. Three checkable values.plan2— six talks into six time slots. Five constraints, including “A is immediately after C”, “F lies strictly between A and B”, “D is later than E”. Exactly one assignment satisfies all. Six checkable values.extrakt2— nine fields from an informal note. With three deliberate traps: three dates, only one of which is the invoice date; a “represented by Mr Berger” who is not the customer; an IBAN with spaces. Plus two deadlines and one gross amount that have to be computed. Nine checkable values.widerspruch— the task with no solution. Five parcels weighing 3, 5, 8, 12 and 14 kilograms across two vans, each van carrying at least two parcels, van A carrying exactly 6 kg more than B, and the 8 kg parcel travelling with B. The correct answer is “no solution” — together with the two weights that would be required. Three checkable values.trace— execution trace. A Python function with a loop, a Fibonacci sequence and a branch on divisibility by three. Two of its return values are asked for. Two checkable values.raetsel— the anchor. The constraint puzzle from the earlier runs, unchanged: five people, five seats, one project and one drink each, eleven constraints. This task alone links the old and new datasets. Fifteen checkable values.
“Fully correct” means every checkable value is right. For trace that is two; for raetsel, fifteen.
The two variants
Task text and format block are word for word identical in both. The only difference:
| Variant | What is added to the prompt |
|---|---|
nackt (“bare”) | Nothing. Only the task and the format block. |
schritte (“steps”) | One sentence before the format block: “Think step by step.” |
The two other variants from earlier runs — the role sentence and the worked analogue — were left out this time. That halved the running time, and it is why everything below concerns only these two versions.
The numbers
Six tasks times two variants times three seeds = 36 cells per model. num_ctx 32768, num_predict 6000, temperature 0, thinking mode off.
| Model | fully correct | mean share | truncated | Speed | Run time |
|---|---|---|---|---|---|
qwen3.5:35b-a3b-q8_0 | 33 / 36 | 0.944 | 0 | 98.4 tok/s | 11.1 min |
qwen3.6:27b | 27 / 36 | 0.778 | 6 | 38.6 tok/s | 32.7 min |
llama3.3:70b | 17 / 36 | 0.595 | 0 | 10.7 tok/s | 50.0 min |
The largest model is the worst and takes four times as long as the best to be so. But the overall figure is the least interesting number in this post. Broken down — fully correct, out of six runs each:
| Task | qwen3.5:35b-a3b | qwen3.6:27b | llama3.3:70b |
|---|---|---|---|
rechnung2 | 6 | 3 | 0 |
plan2 | 3 | 0 | 6 |
extrakt2 | 6 | 6 | 6 |
widerspruch | 6 | 6 | 1 |
trace | 6 | 6 | 3 |
raetsel | 6 | 6 | 1 |
No model wins everywhere. plan2 is the one task the 70B solves flawlessly — and precisely the one qwen3.6:27b fails completely. On rechnung2 it is the other way round. A leaderboard would hide that.
The invented solution
The arithmetic behind the impossible task is short. The five parcels weigh 42 kilograms in total. If A carries exactly 6 kg more than B, then B must carry 18 and A 24. B contains the 8 kg parcel, so 10 kilograms are missing from {3, 5, 12, 14}. No subset of those sums to 10. There is no distribution.
This is what llama3.3:70b answers — in five runs out of six, word for word identical:
ERGEBNIS: LOESUNG; A=17; B=11
17 plus 11 is 28. The parcels weigh 42. The model satisfies the difference constraint — by making 14 kilograms of freight disappear. It did not miscalculate; it silently dropped a constraint in order to be able to answer.
One single run gets it right:
ERGEBNIS: KEINE LOESUNG; A=24; B=18
The two smaller models have no trouble with it: qwen3.5:35b-a3b and qwen3.6:27b both return six out of six — the correct refusal and the two required weights.
This task was written down before the measurement with the expectation that a guessed refusal had to be detectable — which is why the two weights are demanded as well. That held: the 70B gets neither the verdict nor the figures.
The cent that decides between 0 and 3
qwen3.6:27b solves the two-VAT-rate invoice in zero of three runs without the phrase and in three of three with it. That sounds like a rescue. It is a rounding point:
| Variant | Result line | scored |
|---|---|---|
nackt | 19%=1924,69; 7%=100,67; GESAMT=2025,36 | 0/3 |
schritte | 19%=1924,70; 7%=100,67; GESAMT=2025,37 | 3/3 |
One cent. The model had the right method in both versions; the phrase only made it round at the right moment — at the very end, per VAT rate, as the task requires.
By comparison, llama3.3:70b on the same task, where rounding is not the issue:
| Variant | Seed 42 | Seed 43 | Seed 44 |
|---|---|---|---|
nackt | 2058.83 | 2058.48 | 2058.48 |
schritte | 2099.32 | 1486.18 | 1486.18 |
| the correct figure is 2025.37 | |||
With the phrase it does not get better, it gets more unsettled: the spread grows from 35 cents to over 600 euros.
The model that never finished
qwen3.6:27b scores zero out of six on plan2. That is not a wrong answer but no answer at all. All six cells run into the 6000-token ceiling, three of them without ever writing a result line. 16,687 characters, 158 seconds, then a cut mid-sentence. The end of one of those answers, translated:
…often there is only one solution, if one interprets “between” as “not adjacent”? No.
Let us try a different approach:
Perhaps I weighted the direction of “A is after C” and “B is before E” combined with “F between A and B” incorrectly.
If we assume that the time
The model discards its own approach, starts over, discards again. It does not fail at the puzzle, it fails at stopping. llama3.3:70b solves the same task in 504 tokens.
“Think step by step” cuts both ways
Until now this series reported that the phrase hurts on one task in four. That was too simple. Across three models and six tasks it changes the outcome in five places — and in both directions:
| Model | Task | nackt | schritte | Effect |
|---|---|---|---|---|
qwen3.6:27b | rechnung2 | 0/3 | 3/3 | helps |
llama3.3:70b | widerspruch | 0/3 | 1/3 | helps |
qwen3.5:35b-a3b | plan2 | 3/3 | 0/3 | hurts |
llama3.3:70b | trace | 2/3 | 1/3 | hurts |
llama3.3:70b | raetsel | 1/3 | 0/3 | hurts |
In the remaining thirteen places it changes nothing. So the effect depends on model and task, not on the phrase alone. Anyone who adds it to their prompts as a matter of course will sometimes gain and sometimes lose — without knowing which is happening.
The most expensive case is qwen3.5:35b-a3b. Without the phrase it solves plan2 correctly three times:
ERGEBNIS: 9=C; 10=A; 11=F; 12=B; 13=E; 14=D
With the phrase, the same wrong answer three times, at five times the token cost:
ERGEBNIS: 9=B; 10=E; 11=F; 12=C; 13=A; 14=D
It keeps two of the six positions and rotates the rest — not guessing, but a different, consistently carried-through misassignment.
The seeds do nothing, again
In passing, the finding from part four is confirmed once more, and this time unusually plainly, because it is visible in the answer lines themselves. Five of the 70B’s six widerspruch answers are word for word identical. So are qwen3.5‘s three plan2 answers per variant. And with llama3.3 on rechnung2, only seed 42 deviates — the first run, as before, not a particular seed.
What this does not establish
- Not that smaller models are better. The three differ in size, architecture, quantisation and release date all at once. On
plan2the largest one wins. - Not that
llama3.3:70bcannot do arithmetic. It fails at this two-VAT-rate invoice. It solved the simpler invoice from the previous run in eleven of twelve cells. - Not that “no solution” is recognised in general. Exactly one task of that kind was measured. That two models recognise it reliably says nothing about other unsatisfiable tasks.
Three mistakes of my own
First. The 6000-token ceiling was too low. I had raised it from 4000 before the run, explicitly on the grounds that a truncated answer measures my budget rather than the model. It happened anyway — to qwen3.6:27b on plan2, six cells. The zero in the table is counted correctly but read wrongly if taken for a wrong answer. It is no answer at all.
Second. The scoring is all-or-nothing: either every checkable value in an answer is right, or the cell counts as a failure. As a result, an error of one cent looks exactly like a deviation of 539 euros in the table. Had I reported only “0/3 becomes 3/3”, it would have read as a gain in comprehension. It was a rounding point. That difference is not in the metric — it is only in the raw data, and one has to look.
Third. I dropped the rolle and beispiel variants to halve the running time. That was a decision about time, not about knowledge — and it costs an answer: whether a role sentence would have rescued llama3.3:70b on the two-VAT-rate invoice is open. In the previous run, the role sentence was unremarkable exactly where “think step by step” wrecked the extraction task — three of three, the same as with no addition at all. So it did not help; it merely failed to hurt. Whether it would have rescued anything here is precisely what remains unanswered.
The setup
- Two RTX 3090, 48 GB combined, no NVLink. Ollama 0.32.5.
num_ctx32768,num_predict6000,temperature0,thinkoff, seeds 42/43/44. One warm-up run per model, discarded.llama3.3:70bran with 5.1 per cent on the CPU — at this context size it does not fit entirely into 48 GB. The other two ran 100 per cent in graphics memory.- The scoring functions were tested before the measurement against known correct answers, tolerable spellings, the built-in traps, and an answer with no result line. A scorer that fails to recognise the correct answer produces a systematic error across the whole run, and that does not stand out in the data.
What gets measured next: the same six tasks against two versions of the same model that occupy almost identical memory — 48 billion parameters at coarse quantisation against 31 billion at fine, 33.2 versus 33.8 gigabytes. So far that is a matter of belief. It is measurable.