Why One-Shot Benchmarks Measure the Wrong Thing
Most of the coding benchmarks that show up in model cards and leaderboards ask the same question: here’s a task, write the solution in one go. One prompt in, one code block out, compared against a reference answer. It’s convenient to measure and it says surprisingly little about whether a model is any good at real work.
Because that’s not how anyone works. When you fix a bug, you first read your way into an unfamiliar repo, jump between files, run the tests, see which one is red, form a hypothesis, change one spot, run again, correct course. It’s a loop of tool use and feedback, not a single throw. A test that asks only for that one throw measures a skill you almost never use in practice.
I was interested in a practical question: which of my local models should take on the role of the coding agent? The obvious answer would be a code-specialised model. To check that, I didn’t build yet another one-shot benchmark — I built an agentic test that captures the whole loop and scores it in a tamper-proof way. The result flipped my expectation, and the method is more interesting than the result.
The Method: Agentic and Tamper-Proof
The setup is deliberately close to real work. Each task is a small, multi-file repository with a genuine bug and at least one test that fails because of it. The agent gets tools: it can read and edit files, run commands, run the tests itself, and react to their output. It iterates until it thinks it’s done, or until a time limit kicks in. Exactly what a human at the terminal does.
The decisive part is the scoring. If you let an agent grade itself, you’re measuring its self-report, not its work. So the scoring runs in a separate process that the agent never gets to see. And before anything is scored, the test files are overwritten from an immutable copy. So if the agent gets the idea to defang the test instead of the bug, just to turn it green, that change is simply gone again by the moment of scoring. Tampering leads nowhere.
Every result falls into one of five categories, and that split matters more than it sounds:
- PASS — the bug is fixed, all tests green, no regression.
- FAIL — the target test stays red, the problem isn’t solved.
- REGRESSION — the target test is green, but another test broke in the process. The bug was moved, not fixed.
- NO_EDIT — the agent didn’t edit anything at all. That’s not a failure of substance but a tooling or format problem, and it counts as invalid.
- TIMEOUT — the agent ran into the time limit. Also invalid.
This split prevents the most common self-deception in tests like these: mistaking an agent for stupid when it only tripped over the tool adapter, or mistaking a lazy fix for a solution when it broke something elsewhere. A NO_EDIT isn’t a wrong answer — it’s no answer at all, and so it has to be treated differently from a genuine failed attempt.
To keep the test itself from lying, three golden controls run before every pass. The already-correct solution has to come out PASS, the buggy starting state has to come out FAIL, and a tampered version of the buggy state has to come out FAIL as well. Only once those three hold is the verifier trustworthy enough to measure models against. If they fail, it isn’t the model that’s broken but the measurement — and you catch it before you draw wrong conclusions from bad numbers.
Three Difficulty Tiers, Two Models
Two local models went head to head on the same hardware: a general 27B model (Model A from here on) and a code-specialised 30B model (Model B), which up to now had been my default choice for coding tasks. Each task was played twice (n=2), to make gross outliers visible.
Easy: One Bug, a Few Files
Three small repos, each with a clear cause and a single unambiguously failing test.
| Model | Result |
|---|---|
| Model A (general, 27B) | 6/6 PASS |
| Model B (code-specialised, 30B) | 4/6 PASS |
Model B falls behind right here: 4 out of 6. The specialised model fumbles a task the general one solves cleanly.
Medium: The Regression Trap
Six repos across tiers L2 to L4, two of them with a deliberate trap: the naive, obvious fix turns the target test green and breaks another test in the process. Only someone who understands the connection gets past both.
| Model | Result | Failure Pattern |
|---|---|---|
| Model A | 12/12 PASS | — |
| Model B | 2/12 PASS | 4× NO_EDIT, 3× TIMEOUT |
Here Model B collapses: 2 out of 12. And the way it fails is telling. Four NO_EDITs, three TIMEOUTs — the model isn’t solving the tasks wrong, in many cases it never even gets to editing anything useful or finishing in time. Model A goes through the entire tier without a single dropout: 12 out of 12.
Hard: Cause and Symptom in Different Files
Three larger subsystems — a bytecode VM, a dependency resolver, and a JSON query engine. Multi-stage bugs where the wrong result surfaces somewhere entirely different from its cause.
| Model | Result |
|---|---|
| Model A | 6/6 PASS |
| Model B | 0/6 PASS |
On the hardest tier Model B tips over completely: 0 out of 6. Model A holds here too: 6 out of 6.
Adding up all three tiers, Model A stands at 24 out of 24 PASS, Model B at 6 out of 24. Model A took between 20 and 70 seconds per task and produced not a single NO_EDIT and not a single timeout.
A Fix at the Root, Not the Symptom
One run shows especially clearly where the difference lies. In the hard tier, the bytecode VM held a bug that showed up as a wrong arithmetic result — at a spot in the code that had nothing to do with the actual cause. The obvious, local move would have been to correct the result right where it came out wrong. That exact move would have broken another test and failed as a REGRESSION.
Model A went back to the source instead. It found that a jump instruction was misclassified in the opcode table. A single entry in the wrong category, and under certain conditions the VM branched incorrectly — with a symptom far away from the fault. The fix sat in the table, not at the place where the wrong value appeared. That’s the difference between “make the red line green” and “understand why it’s red.” The tamper-proof verifier exists precisely to keep those two things apart — a symptom-patch would have been caught here by the regression trap.
The Honest Limits
Now for the part most benchmark posts leave out. These repos are curated. Every task has one cause, a clearly failing test, and few files. That’s on purpose, because it’s the only way a clean, tamper-proof scoring is possible — but it also means this ladder sits one rung below real, large open-source code. A real repo often has several causes at once, unclear or missing tests, historically grown baggage, and files numbering in the hundreds.
So the fact that Model A doesn’t tip over on any of the three tiers does not mean “it solves arbitrary repos.” It means exactly one thing: this model’s ceiling lies above this ladder. How far above, the test doesn’t say — that would take harder tasks, which are then harder to score cleanly. That’s the price of tamper-proofing, and I’d rather name it openly than hide it.
And the sample is small: n=2 per cell. Two runs per task catch gross outliers, but they aren’t statistics. The value of this test isn’t in a large number of runs but in the method — separate process, immutable test copy, clean categories, golden controls before every run. An honest small measurement is worth more than a large one that lets itself be fooled.
Code-Tuning Wasn’t the Missing Ingredient
That leaves the question I started with. I’d expected the code-specialised Model B to win — that’s what it’s tuned for, after all. Across all tiers it stands at 6 out of 24, the general Model A at 24 out of 24. And the gap grows with difficulty: easy 4 against 6, medium 2 against 12, hard 0 against 6.
The lesson is uncomfortable for a widespread gut feeling. More “coding tuning” wasn’t the missing ingredient. What makes the difference isn’t specialist knowledge about syntax, it’s the ability to sustain the loop: read the repo, form a hypothesis, drive the right tool, learn from the test result, and reach for the cause instead of the symptom. That’s exactly where Model B visibly fell down — four NO_EDITs and three timeouts in the medium tier alone aren’t a knowledge problem but a craft problem.
For me that has a concrete consequence: the default choice for the coding agent is no longer the specialised model but the general one. The raw data for every run is out in the open at bench.tobiasliauw.de. Anyone who wants to rebuild the method or doubt the numbers will find everything they need there.