Notes from building an evaluation harness for entity linking against a hierarchical vocabulary
Most published model comparisons answer a question I don't have. They tell me a model scores well on reasoning benchmarks, or emits valid JSON 97% of the time, or runs at 130 tokens per second. Useful context. But when I needed to know whether a locally-hosted model could take an unstructured document and attach the things it mentions to the right node of a controlled vocabulary — the kind of task that sits underneath a lot of unglamorous production work — none of those numbers moved the needle.
So I built a small harness and pointed it at seventeen models running on two consumer GPUs. What follows is mostly what surprised me, including several occasions where the thing I'd built was wrong and the models were fine.
Precision and recall hide the failure that matters
The task: read a document, extract the entities, and link each one to a node in a hierarchy. The hierarchy has depth — broad families at the top, narrowing through intermediate layers, down to specific named leaf nodes at the bottom.
The obvious metrics are precision and recall against a gold set. They're not wrong, they're just blind to the one thing you actually care about. If a document clearly evidences a leaf node, and the model links the root instead, precision and recall both call that a hit. It found "a correct entity". But collapsing a deep hierarchy into its top node is exactly the failure that makes the vocabulary pointless — it's the difference between a searchable structure and a bag of tags.
So the harness scores two failure modes separately:
- Over-generalisation — linked an ancestor where the document supported something deeper. The flattening failure.
- Over-specification — linked a specialisation the document does not support. The inventing failure.
These are not symmetric. Over-generalisation makes the output less useful. Over-specification puts a claim in front of a human reviewer that the source never made, which is worse: it costs trust, and trust is what makes anyone read the second batch of suggestions.
Splitting them changed the ranking. Two models with near-identical recall separated cleanly once I could see that one was quietly flattening and the other wasn't.
Ancestors are implied, and demanding them punishes good behaviour
First version of the gold data listed the whole ancestor chain. If a document evidenced a leaf, the gold expected the leaf and every node above it.
That's wrong, and it took a confusing result to see it. A model that links the leaf has already asserted the ancestors — the hierarchy encodes that. Demanding it emit them anyway penalises precisely the deepest-node behaviour I'd asked for in the prompt. My best models were being marked down for being correct.
Ancestry is now implied for recall. Not for precision, though: implied nodes earn recall credit but must never enter the precision numerator. I know this because letting them do so produced a precision of 1.12, and an impossible number is a gift — it announces the bug instead of quietly shifting a ranking.
The empty response that looks perfectly healthy
The failure that cost me the most time: a model returns nothing, and every metric you'd normally check says it's fine.
Reasoning-capable models spend their token allowance on an internal trace. If the allowance runs out before the trace finishes, you get a response containing zero characters — while the server cheerfully reports a healthy tokens-per-second and a full token count. If your harness scores structure rather than content (counting tool calls, say, or checking that a parse didn't throw), an empty answer sails straight through as a pass.
In one earlier dataset I went back and checked, 36% of responses in a category were zero bytes, and the failure had been invisible for weeks because the throughput column looked normal the whole time.
The fix is trivial and belongs in every harness: assert the response is non-empty, and treat empty as a distinct failure class rather than a parse error. The diagnosis is what matters — "returned nothing" and "returned malformed output" have different causes and different fixes.
I was measuring my own budget, not the models
Related, and more embarrassing. I'd set a token ceiling that felt generous. Several models failed most of the suite.
Then I noticed every single failure landed on exactly the ceiling, with zero characters, while every success finished comfortably below it. That's not a distribution of model capability. That's a wall I put there.
Raising the ceiling took one model from one usable result out of six to six out of six at perfect recall — and it was a model I had already written off in my notes as "unsuitable for this task". It was suitable. My budget wasn't.
The general lesson: when failures cluster exactly on a parameter you chose, suspect the parameter before the subject. A histogram of where failures land is a thirty-second check that would have saved me a wrong conclusion.
A second pass: the scorer moved the ranking, the bigger context window didn't
The first sweep had entangled two things I wanted apart: how much of the ranking came from what the models did, and how much from what my scorer made of it. So I took one model and a single task and ran it five times per context window — the same small window and a very large one, the same five seeds, at the sampling preset. Five draws per setting, because this one ran at temperature one; a single run wouldn't be a measurement there.
The first result isn't about the models. It's about me. I re-scored the exact same answers — byte-for-byte the same files, nothing regenerated. The old scorer passed two of the five seeds; the new one passed none.
That's my rule moving under a fixed input, not the model. When I diffed the two scorers, six of seven feature patterns had shifted — three tightened, three loosened.
To find what actually caused the collapse, I changed one rule at a time from the old scorer (baseline: four passes across ten moves, both windows). Tighten only the temperature regex: two of ten. Drop only the free point for a control: two of ten. Both together: zero of ten. No single change tips it to zero — each halves the pass rate on its own, and only the pair reaches nothing. An earlier note of mine claimed a tightened regex alone flipped no move at all. That was simply wrong.
The weakest move in the field shows why. It computes buoyancy, cooling, temperature and steam — and scores zero on all four, because its fields are named "temp", "buoy", "coolingRate" and "nSteam", four names my checker doesn't know.
The physics was there. My regex wasn't. It's the same failure shape as the compliance check further down: a check punishing correct behaviour because it can't recognise it.
The new scorer ships two golden controls — one file that must score 16 out of 20, and one that must fail at 1 out of 20. The old scorer had none. But it's worth being clear about what they do and don't do: they prove the scorer isn't broken. They didn't catch this bias, because they use the canonical field names. I only caught it by reading the artifacts by hand.
The second thing I wanted to isolate was the context window. Across the two windows the mean score was 12.8 against 12.6 out of twenty, median runtime 83.7 against 84.1 seconds, median token count 5251 either way. Three of the five seed pairs came back byte-identical — so, effectively, two independent samples per cell. The 0.2-point gap is a single length point one seed tipped across a threshold. I didn't find a difference beyond VRAM in this setup — and with five draws per cell, three of them identical, a small one is something I couldn't have seen anyway. That's not equality; it's a non-difference.
Those five seeds ran twice, once per window. The ten moves are published: prompt, response, thinking trace, score and the generated artifact are all there, so the scoring is yours to re-run. Three numbers aren't in them, though: runtime, token count and the reason a move ended live only in my run log, not in the per-move score files. If those are the three you want to check, the published artifacts won't let you. The ten exemplars are here.
An architecture that takes the server down
Four of the models I tried aborted the inference server outright — not a bad answer, a crashed process, roughly nineteen seconds burned per call.
The interesting part was the attribution. This crash had previously been recorded as a consequence of schema-constrained decoding. My harness doesn't use constrained decoding at all, and reproduced it exactly. The real trigger was a specific model architecture interacting with the runtime's context-checkpointing, visible in the server log as a checkpoint being created immediately before a CUDA error.
Two things follow. First, a cheap architecture check before loading a model is worth having — you can query it in a fraction of a second and skip a model that will only waste twenty seconds per call. Second, the ban has to be arch-exact. The dense variant of the same model family, one character different in the architecture string, runs perfectly and produced two of the four best results in the whole sweep. A slightly-too-broad exclusion rule would have thrown away top performers.
A compliance check that fired on correct behaviour
The harness includes a rule-compliance check: the prompt forbids the model from emitting judgements of its own, and the harness independently verifies it complied. Reasonable.
One of my test documents deliberately contains self-assessment language, because a document in the wild might. The prompt also requires every extracted entity to carry a verbatim quote from the source as evidence.
You can see the collision. A model that correctly quoted the source as evidence tripped the compliance check — and it was flagged while scoring perfectly on the same document. The check punished the exact behaviour the contract demanded.
The fix is to excise quoted spans before scanning, so the check flags the model asserting something in its own voice rather than faithfully reproducing the input. Structural checks still apply: a model that adds a judgement field to its output is asserting, not quoting, and that should always fire.
The wider point is worth more than the fix. A compliance check with a false-positive path is worse than no check, because it produces confident wrong verdicts that steer you away from good options on invented grounds. It nearly cost me the best model in the field. Any automated policy check needs its own positive and negative controls, exactly like a measurement does.
I also learned to store where the check matched. A later model tripped the same rule with a perfect score, and I couldn't tell a genuine violation from another false positive because I hadn't kept the matching text. A verdict you cannot audit isn't a verdict.
Determinism is worth confirming, once
At temperature zero, two consecutive full passes were byte-identical — same scores, same per-document results.
That's a small check with a large payoff. It means a single run is a trustworthy measurement — as long as you stay at temperature zero — and that any difference I see between two runs is a real change in the system rather than sampling noise. Without it, every comparison needs repeated trials and error bars, and the cost of the whole exercise multiplies. Run it once, write down the result, stop wondering. That reliance only holds at temperature zero, though: a later re-probe ran deliberately at temperature one, and there a single run stops being a measurement — it needed five draws per cell.
What the numbers ended up saying
The headline is unremarkable and probably generalises: the fastest model was not the best one, and the biggest model was not the best one either.
The winner was a mid-sized model at roughly 94 tokens per second — slower than several alternatives running at 130+, and smaller than the largest candidates. It won because it was the only model combining perfect recall, zero flattening failures, and completion on every document. Three other models also hit perfect recall but each ran one document into the token ceiling — a move the harness now records as invalid under its own rule, not as a failure of the model. The winner's edge isn't one of capability; it's one of complete measurement: it stayed inside the budget on every document.
For a workload where a human reviews the output, that trade is obvious once you can see it: throughput is nearly free and a flattening failure is not. But you can only see it if your metrics distinguish the failure modes. On a generic scoreboard, four of these models look interchangeable.
The other durable result: several models did the job well enough. The gap between the top few is smaller than the gap between a good harness and a bad one. I spent far more time fixing my own measurement than choosing between models — four of the bugs I found were mine, and each one would have produced a confident, wrong recommendation.
If you're building something similar
- Score the failure modes your domain actually cares about, not the ones that are easy to compute.
- Check where failures cluster before concluding anything about capability.
- Assert non-empty output. Always.
- Record token-ceiling aborts as invalid moves, never as a failure of the model.
- Give every automated check — including policy checks — a positive and a negative control.
- Keep the evidence for any verdict you might later need to dispute.
- Confirm determinism once, and trust single runs only at temperature zero.
- Test the architecture before you spend twenty seconds per call finding out.
None of this is sophisticated. All of it was necessary, and I got most of it wrong the first time.
Live examples: the same discipline, applied to game frontends
A follow-up benchmark asked local models to build casino presentation clients — a slot and a crash game where the server owns every outcome and the client is only allowed to render it: no Math.random, money as integer minor units, the grid taken verbatim from the server. Below are unmodified model outputs, embedded in a sandboxed frame on an isolated origin. Spin and cash out yourself.
A first-shot pass (TeichAI Fable-Distill, PixiJS)
One draw, no repairs: 9/9 contract checks. It also spent 85,000 characters thinking before writing a byte — quality and latency are different axes, which is the point of the whole article.
A crash client that survives its own contract (qwen3.8, Phaser 3)
Cash out mid-round and the payout is computed in integer cents from the server tick stream; after the crash the button is dead and the payout immutable.
Contract-green and still wrong — the failure mode a verifier cannot see
This one passes every mechanical check after one repair round. Look at it. The layout is clipped into a corner — a defect no contract assertion catches, which is why the pipeline ends with a vision gate and not with a green checkmark.
All six examples, including plain-DOM and pure-canvas variants: the full gallery.