In July I published an article about determinism in local models. It contains this sentence: “All 198 outputs are available in full, so that you can contradict me.”
That is no longer true. The second machine from that comparison was wiped in early August for a reinstall. The outputs lived there. There was never a download link, because I assumed “I have them” was enough.
A benchmark whose evidence nobody can check ends up being a claim with a table next to it. I only noticed when I went looking for the files for a new comparison and found they were no longer there.
The second finding was worse
While cleaning up, I noticed my current test rig does the same thing. It asks models to build an interactive simulation as a single HTML file, checks the result mechanically — syntax, animation loop, whether certain physical quantities appear in the code — writes one line of measurements to a log, and throws the generated file away.
The result looked clear-cut: one model passed 5 of 5 runs, another only 2 of 5. Recomputing from the raw data told a different story. The weaker model had produced syntactically valid, animated code in 10 out of 10 runs. The entire difference rested on a search for words in the source. Name the buoyancy force F_b instead of buoyancy and you fail.
Whether that is what actually happened, I can no longer determine. The files are gone.
A scoring system that discards the thing it scored makes its verdict irrefutable — and therefore worthless.
The ceiling that looks like incompetence
There was a third finding, and it annoyed me most because it would have been so easy to avoid. The rig gives each model a token budget. I had set 8192, which felt generous. But that budget pays for more than the answer — it also pays for the thinking that comes first, and for models that reason before writing, that is the larger item.
One model in the field thinks at length. In one run it produced almost 21,000 characters of reasoning before the first line of code existed. In three runs out of five it hit the ceiling exactly: the file broke off mid-sentence, was therefore not valid JavaScript, and my scoring dutifully recorded zero features, invalid syntax, failed.
The model landed at 2 out of 5 and looked like the weakest entrant. When I recomputed only the runs it had been allowed to finish, the record read 2 of 2 passed, both with full feature coverage — the highest in the entire field, higher than the model that had topped the table.
Think hardest, lose. That is the precise inversion of what the test was supposed to measure. And nothing in the output said so: “failed” and “truncated” looked identical in my log. Since then a run that hits the token limit is invalid and drops out of the average. It is not a bad result, it is no result.
What I built instead
Three requirements, in this order:
- Every artifact is kept — the generated file, the model’s complete response, the reasoning trace, the exact prompt, the individual sub-scores, and a checksum.
- It has to run in the browser. Showing an interactive simulation as a screenshot is an admission of defeat. You have to be able to click it.
- The link has to stay stable. Content-addressed storage: same file, same key.
Point two is the uncomfortable one. Because it means: unreviewed, machine-generated JavaScript is going to execute in my readers’ browsers.
Counting words is not checking physics
The feature check searched the source for terms: does “buoyancy” appear, does “drag” appear, does “groundwater” appear. So that a model cannot simply copy the list into a comment, I strip all comments and string literals first. That was well intended and had a side effect I saw too late: what remains is pure code, and so what gets checked is the vocabulary of the variable names.
For one model the term “buoyancy” was missing in nine runs out of ten. Nine out of ten. When an ability is genuinely absent, that fluctuates; a nine-in-ten failure on exactly one word does not look like inability, it looks like somebody consistently calling the buoyant force something else. F_b, for instance, as physics usually does. Or lift.
Whether that was the reason, I cannot say. By then the files were already gone. That is the point where an annoyance turned into a consequence: a verdict whose subject can no longer be examined can neither be refuted nor defended.
I kept the check itself, but weighted it differently. A term now scores two points if it appears on a line where something is also computed — an assignment, a multiplication, a call to a maths function. If it merely sits there, it scores one. That separates “implemented buoyancy” from “named a variable that” without insisting on a particular word.
Why that is not a detail
Nobody reads through ten generated files. At a hundred, certainly not. It would also miss the point. I want to show what came out, not pick the three prettiest.
If that code sat on the same origin as the blog, a generated script could read visitors’ session data. Not because a model would be malicious, but because nobody is looking. Three layers, all of them necessary:
- A separate subdomain, its own vhost, server-side scripting turned off.
- A content policy with
connect-src 'none'. That is the decisive line: even a generatedfetch()cannot reach the outside.script-src 'unsafe-inline'cannot be avoided, because the artifacts are inline scripts themselves. Hence the separate subdomain. - In the article, a frame with
sandbox="allow-scripts"— withoutallow-same-origin. That combination gives the frame an opaque origin: the script runs and animates, but reaches neither cookies nor storage nor the parent page. Both attributes together cancel the protection. That is the trap you fall into once.
Three things that went wrong
All three looked like success until I measured.
The object store did not start, but reported no error. The container image passes configuration files through differently than expected; the credentials were not readable by the main process. The service died with a fatal, the container restarted, four times. The odd part: a test read inside the container could open the file without trouble. Check only that, and everything looks fine.
The protective policy was missing exactly where it counts. The web server set it correctly in the outer block. For the artifacts themselves I had a separate block that only governed caching. But add_header is not additive: such an entry in the inner block replaces every inherited one. The configuration read perfectly. The response for the artifact file carried no policy. I only caught it because I fetched the headers of a real file instead of reading the config.
The store had an invisible ceiling. The default permits eight storage units. That is enough, until it isn’t — and then every write fails. I had seen the same failure two days earlier somewhere else. It is now a comment in the configuration file.
The rule I take from this
A passing test says very little unless there is a counter-test beside it that is known to fail. Without the second one you have only checked that something happens at all.
So the rig now has two hand-written control files. One must score full marks. The other contains all the terms being searched for — but only inside a comment — and must fail. If both behave before every measurement, it is proven that the scoring reads code and not commentary.
On the first run that gave 16 out of 20 for the good file and 1 out of 20 for the bad one. Only then was a model allowed to compete.
What it costs
The storage is smaller than you would think. An artifact runs to 15 kilobytes at the median, the largest in the field so far to 40. A complete run of thirty draws comes to roughly half a megabyte. A thousand such runs would be just under 500 megabytes — for an archive backing every published comparison, that is nothing.
I did wonder at one point whether this justifies its own distributed object store, and the answer at first was a clear no. At that volume it would be overhead. That one is running anyway has a different reason: the access contract. If several machines are to write artifacts, an endpoint with credentials is cleaner than a network share plus a copy step. Content addressing helps too: files identical across runs — the same images, the same templates — land under the same key and are stored once.
The real effort sits elsewhere. It sits in getting every step wrong once and noticing. Three of the mistakes above cost more time together than the whole rest of the build, and all three had the same shape: a configuration that looked right, and a result nobody checked.
Why I am writing this down at all
There is a comfortable version of this text. In it I build the rig, show the numbers, and leave out the three mistakes because they are fixed anyway. That version would have looked better and been worth less. The interesting part of a measurement is rarely the result; it is the question of how you would have noticed it was wrong.
For all three mistakes the answer is the same: you would not have, not if you only look where you were already looking. The object store reported itself as running. The web server configuration read correctly. The storage ceiling was written down nowhere. In all three cases it took a call from outside to expose the gap — fetching a real file and reading its headers instead of reading the config; asking the service a question instead of asking for its status. That sounds banal until you notice how often you do the opposite because it is quicker.
The same thing in miniature: I wrote a control file for the rig that must fail. It contains every term being searched for, but only inside a comment, plus a deliberate syntax error. If the scoring gives it points, it is reading commentary instead of code, and every number beside it is worthless. Together with a second file that must score full marks, both run before every measurement. On first use that gave 16 out of 20 for the good file and 1 out of 20 for the bad one. Only then was a model allowed to compete — and I knew the result was at least measured rather than guessed.
That cost an afternoon. What the absence of that control cost me was three false claims in a text I very nearly published as it stood.
What this means for the older articles
The numbers stand. What does not stand is the sentence claiming the outputs are available in full. I am not letting it quietly disappear — it gets a dated correction.
From the next run onward, every artifact is public, clickable, with the prompt and the complete response next to it. Anyone who wants to contradict me should be able to, without having to ask.