Exit code 0 is not evidence — six nights of a benchmark that measured nothing
A benchmark ran every night for six nights, reported success every time, and measured nothing at all. The systemd unit exited cleanly. The result files were…
Blog
Infrastructure, AI, tools, and the decisions behind them.
A benchmark ran every night for six nights, reported success every time, and measured nothing at all. The systemd unit exited cleanly. The result files were…
Notes from building an evaluation harness for entity linking against a hierarchical vocabulary Most published model comparisons answer a question I don't have. They tell me…
Why One-Shot Benchmarks Measure the Wrong Thing Most of the coding benchmarks that show up in model cards and leaderboards ask the same question: here’s a…
Thirty-six runs, four model families, four task designs — and three of my own explanations dying along the way I wanted a number. Can the models…
I work with several machines and several AI sessions at once. One on the server, one on the Windows box, occasionally a third somewhere in between.…
In July I published an article about determinism in local models. It contains this sentence: “All 198 outputs are available in full, so that you can…
Twelve models, one structured-extraction task, and a challenger with the better argument on paper The companion piece to this one reports a zero: local models, driven…
Yesterday this blog reported that the test set had hit its ceiling: one model solved all 48 cells, and about a model that solves everything nothing…
Part 8 closed with a promise: next I would measure what abliteration actually takes away from a model. The advice itself is everywhere — take the…
Until today, every measurement in this series ran on a single graphics card. 24 gigabytes, and the boundary was clear: anything larger simply went unmeasured. As…