Thirty-six runs, four model families, four task designs — and three of my own explanations dying along the way
I wanted a number. Can the models I run on my own hardware do real programming work — not exercises, but the kind of change you make in a codebase that already exists, with tests that already pass and must keep passing?
The number turned out to be zero. What made the exercise worth the two days was everything I had to discard to trust it.
The setup, and why it is built the way it is
A coding agent is a program that gives a model tools — read a file, edit a file, run a command — and loops until the model says it is finished. I used two of them, driving models served locally on two consumer GPUs.
The scoring rule matters more than anything else here: the model is never scored on its own report. Verification runs in a separate process, against the project’s own test suite, and the model never sees it. This is not paranoia. Early on, one agent returned a success code in five seconds along with the sentence “Both files have been created.” It had written nothing.
There is a second rule that turned out to be just as load-bearing. The fix must not break anything that already worked. The task is not “make this test green” — it is “make this test green while twenty-two others stay green.” That constraint is what separates a real coding task from a puzzle, and it is exactly the constraint an exercise cannot have.
Four tiers, each removing one excuse
I built the task four ways, so that a failure would name its own cause.
Tier 1 is textbook exercises in an empty directory: convert a number to Roman numerals, parse a duration string, fix a two-line bug in a ten-line function. Tier 2 is a genuine defect in a repository of several hundred source files, with nothing but the failing test’s name to go on. Tier 1.5 is the same defect, but the file to change and the test are pasted into the prompt in full — no searching required. Tier 1.6 removes the agent entirely: one direct call to the model, which must return the complete corrected file, which I then write to disk myself.
Tier 1 scores three to four out of seven. Everything else scores zero.
The three explanations, and how each one died
My first explanation was navigation: hundreds of files is too much to search, the model gets lost before it gets to work. Tier 1.5 exists to test exactly that, and it refuted it flatly. Handing over the file changed the rate at which models wrote anything at all from two in sixteen to two in sixteen. Not a small improvement — no improvement.
The second explanation was better, because I found it in the logs rather than in my head. One agent has a loop-detection guard: after repeated failing edits it asks for permission to continue, and in a non-interactive run there is nobody to grant it, so it silently declines its own request. Every measurement I had taken with that agent was measuring a permission dialogue. That is a real defect and it had to go. Removing it moved the write rate from two in sixteen to three in sixteen.
The third explanation was the most interesting, and the one I most expected to hold. Editing an existing file means emitting a precise search-and-replace: find this exact string, put that one in its place. Small models are known to be shaky at exact reproduction. Writing a whole new file is a different operation — and tier 1 shows they can do that. So: let them write the whole file.
Tier 1.6 gave every model the complete source, the complete test, and one instruction — return the entire corrected file in a single code block. Zero out of four.
The artifact that ended the argument
One model returned a clean, complete file of the right length. Sixty-nine lines in, sixty-nine lines out. I diffed it against the original. Here is the entire change it made:
- a comment line beginning with
#now begins with//
That is all. It did not touch the bug — the fix does not appear anywhere in the file. What it did do was replace a Python comment marker with a JavaScript one, which turns the module into a syntax error. The model reproduced three hundred lines of context flawlessly and did not perform the one reasoning step the task consisted of.
A second model behaved the same way with a different wrong answer. A third returned a hundred and twenty-five bytes of prose where a file was asked for.
The fourth model, and why an empty answer is not a result
The fourth returned nothing at all, and I nearly filed that as a failure. It is not one — an empty response is a broken measurement until you know why it is empty.
Measured natively, the picture was this: 31,553 tokens of internal reasoning, zero characters of answer, and a completion reason of “stop” — the model believed it was done. Reading the reasoning trace is instructive. It circles the right function twenty-three times and never reaches the diagnosis. Toward the end the text visibly decays: the same identifier appears in four different misspellings, whole paragraphs repeat verbatim, and the last lines are fragments.
I had a theory: the repetition penalty, set fairly high, punishes reuse of tokens the model needs, and over thirty thousand tokens that drives it into misspellings. Testable, so I tested it. Setting the penalty to neutral made it worse — the reasoning trace grew from a hundred and twenty-seven thousand characters to two hundred and fifty thousand, still with no answer.
Disabling the internal reasoning did produce output: two hundred thousand characters of it, containing fifty-five separate code blocks. I compiled all fifty-five. Not one is a valid, complete version of the file.
What I got wrong, which is most of the method
I claimed at one point that prompt processing on this hardware ran at about a hundred tokens per second, which would have made agentic work impossible on arithmetic alone. It was wrong. I had divided wall-clock time by token count on a call that included loading a nineteen-gigabyte model from disk. The actual figure across nearly four thousand samples is around a thousand tokens per second for realistic prompts, and the per-call overhead on a one-token prompt reads as eighty-eight — a number that looks like a throughput measurement and is nothing of the kind.
I then blamed a different process for saturating the hardware, and produced a control to prove it: watch the load while nothing of mine is running. The load continued, so it was not mine. Except my process-matching pattern did not match the process I was looking for, which runs under a different name. The load was mine after all.
I reported that a configuration change had eliminated a specific warning from the logs, having checked a three-minute window in which nothing happened that could produce the warning. Zero occurrences, no positive control, no meaning. It fired twenty-five more times.
Each of these was caught by the same cheap habit: before believing that something is absent, prove your test can see it when it is present. Every time I skipped that step I produced a confident, well-argued, wrong answer — and each of those answers was on its way into a decision.
What the zero is worth
The result is narrow and I want to keep it narrow. It says that models of this size, driven by these agents, do not perform the read-change-test-repeat loop on an existing codebase. It does not say they are useless: the same models pass half the exercises, and on a structured-extraction task I run regularly, one of them achieves perfect recall with no errors of the kind that task punishes.
The honest shape of the finding is that these two things are far apart. Generating code into empty space is a capability these models have. Changing code that already exists, under the constraint that everything else keeps working, is one they do not — and no amount of prompt engineering, permission-flag fixing or output-format redesign moved that boundary by a single test.
What did move was my confidence in three explanations that felt right and were not. That, in the end, was the more expensive lesson.
Addendum: the fourth explanation I did not have then
Three explanations died in this piece. A fourth has since joined them, and it is more uncomfortable than the first three, because it is not about the model. It is about me.
I have since built a different rig, one that has models write an interactive simulation as a single file. It produced a clear ranking too: one model passed five runs out of five, another only two. When I recomputed from the raw data instead of trusting the table, nothing of that ranking survived.
The weaker model had delivered syntactically valid, running code in ten runs out of ten. Every single time. The entire difference rested on a search for terms in the source — and because I strip comments and string literals first, that search ends up checking the names of the variables. Call the buoyant force F_b, as physics usually does, and you fail. For one model exactly one term was missing in nine runs out of ten. Nine out of ten on a single word does not look like a missing ability.
A second effect came on top, and it is worse. Each run had a token budget, and that budget pays not only for the answer but for the reasoning that precedes it. One model in the field reasons at length — in one run almost twenty-one thousand characters before the first line of code existed. In three runs out of five it hit the ceiling exactly, the file broke off mid-sentence, and my scoring dutifully recorded invalid syntax, zero features, failed.
The table then read two out of five. Count only the runs it was allowed to finish and it reads two of two passed, both with full feature coverage — the highest in the field. Think hardest, lose. That is the precise inversion of what was supposed to be measured, and in the output “failed” and “truncated” looked identical.
What does that mean for the zeros in this article? I went back through those runs, and the answer is: the zeros hold. Nothing was truncated there, and the scoring ran against executed code rather than variable names. The finding stands.
What does not stand is the ease with which I wrote it down. Checking three explanations for whether the model failed, and none for whether the measurement failed, was a gap — and the fact that I only noticed it at a different rig does not speak well for the care taken at the first one. Since then two hand-written control files run before every measurement: one that must score full marks, and one that must fail, because it carries every searched-for term inside a comment only. If the second one scores, the grading is reading commentary instead of code, and every number beside it is worthless.
That cost an afternoon. Without it I would have published three false claims in another piece, and they would have looked exactly as convincing as the table here.