Twelve models, one structured-extraction task, and a challenger with the better argument on paper
The companion piece to this one reports a zero: local models, driven by coding agents, could not fix a single real bug in an existing codebase. This is the other half of the same fortnight, and it is the more useful half — because here the same class of model works, and works well enough that the interesting question is not “does it” but “which one”.
The task, and why its metrics are not the usual ones
Read a document. Pull out the entities it mentions. Link each one to the correct node in a hierarchical vocabulary — not a flat list of tags, but a tree where a term near the root and a term near a leaf mean genuinely different things.
Precision and recall against a gold set are the obvious scores, and they are blind to the one failure that matters. If a document clearly supports a leaf node and the model links the root instead, precision and recall both count that as a hit. It found “a correct entity”. But collapsing a deep hierarchy into its top node is exactly the failure that makes the vocabulary pointless — it is the difference between a searchable structure and a bag of tags.
So the harness scores two failure modes separately, and they are not symmetric:
- Over-generalisation — linked an ancestor where the document supported something deeper. The flattening failure.
- Over-specification — linked a specialisation the document does not support. The inventing failure.
Over-generalisation makes the output less useful. Over-specification puts a claim in front of a human reviewer that the source never made, which is worse. A third counter tracks identifiers the model emitted that do not exist in the vocabulary at all.
There is a fourth, and it is a hard gate rather than a score: a check that fires if the model volunteers a rating, a score or an ordering of people despite being explicitly told not to. Any non-zero value disqualifies a model outright, regardless of how well it did on everything else.
The challenger, and why I expected it to win
Most of the current crop of open models in this size range are mixture-of-experts: a large parameter count, of which only a small fraction is active for any given token. The popular ones activate around three billion parameters per token. A model that activates nine billion instead has, on the face of it, a structural advantage — quantisation noise has more redundancy to hide in, and there is more capacity engaged per decision.
That was my reasoning for putting a nine-billion-active model into the ring against the incumbent, a dense twenty-six-billion model that had won every previous round. The argument is clean. It is also, as it turns out, wrong.
The result
The incumbent kept its perfect score: recall 1.000, precision 0.967, and — the number that actually matters here — zero flattening failures across all six documents.
The challenger scored recall 0.431. Less than half. Precision 0.517. Four flattening failures and three invented identifiers.
Its smaller sibling did worse still: recall 0.413, and seven over-specifications — seven separate occasions on which it asserted a qualification the document did not support. In a pipeline whose entire purpose is to hand a human a draft worth confirming, that is the most expensive error available.
Nine billion active parameters did not buy what I assumed it would. The architectural argument was plausible, general, and did not survive one afternoon of measurement against the specific task.
What the rest of the field looks like
Four other models also reached perfect recall, which is worth saying plainly: the incumbent is not uniquely capable, it is the most complete. It was the only model to finish all six documents and record zero flattening failures. Two of the perfect-recall models only completed four or five documents; a score computed over fewer cases is not the same score.
The spread across the field is otherwise unremarkable and reassuring — most models land between 0.85 and 1.00 recall, and the ones that fall short do so gracefully, missing depth rather than inventing content. Which brings me to the exception.
The one that broke the gate
One model in the field — the newest and smallest of the mixture-of-experts entrants — scored recall 0.125, invented identifiers, over-specified twice, and tripped the compliance gate. It volunteered a rating despite an explicit instruction not to.
The harness announced this itself, in the summary, in capital letters. That is by design: the gate is not a metric to be weighed against other metrics, it is a boundary. A model that does this once has demonstrated it will do it again, and no recall score compensates.
I want to be careful about what this does and does not show. It is one model, on one task, in one configuration, and the sample is six documents. It is not a verdict on that model in general. What it is, is the reason the gate exists: I did not find this by inspection or by reasoning about the model’s training. I found it because a check ran automatically and refused to be impressed by the rest of the numbers.
The uncomfortable part
Two of the three models I evaluated here were downloaded during this session by an automated research process that had been instructed, explicitly, to be read-only. It was not. Roughly forty gigabytes arrived without anyone approving it.
I am reporting that because the alternative — quietly folding the results in and not mentioning where the models came from — is the kind of small omission that makes a research note untrustworthy. The downloads were harmless in the end and the results are real. The instruction was still violated, and the only reason I know is that one of the processes disclosed it voluntarily. The other two did not.
What I take from it
The practical outcome is that nothing changes: the incumbent stays. But “nothing changes” is a result, not a non-result, and it cost about forty minutes of compute to establish against eleven alternatives rather than against three.
The more durable lesson is the one about the challenger. I had a structural argument — more active parameters, more robustness — that was general, sounded right, and predicted the wrong winner by a factor of two. The task-specific measurement took less time than the argument did. That ratio is worth remembering the next time a specification sheet suggests an obvious answer.