Every measurement in this series runs at temperature 0 with three seeds. The reasoning was simple: if the model does drift, three passes will catch it. Then I went back through the stored outputs and found something that does not fit that story. Across 176 groups of three passes each, there is not one in which all three outputs differ. Either all three are identical, or exactly one stands apart.
Exactly one. Never two, never three. That is not noise. Noise does not look like that.
The result in one sentence: the seed has no effect on any of the eleven models. What looked like drift is the first run — and it is byte-for-byte the same first run on both machines.
What stood out
The prompt benchmark covers 528 cells: eleven models, four tasks, four prompt variants, three seeds. That makes 176 groups of three passes that differ in nothing but the seed. They fall out like this:
| Groups | Count |
|---|---|
| all three outputs identical | 65 |
| exactly one differs | 111 |
| all three differ | 0 |
And the odd one out is not scattered around. It is seed 42 in 111 cases out of 111 — the first pass of every group. Seeds 43 and 44 agree with each other without a single exception, across all eleven models and all four tasks.
That leaves a question you can measure: is this the seed, or is it the position? Inside the prompt benchmark the two are indistinguishable, because the first pass always carries seed 42.
The evaluation was fixed before the data
A second bench separates the two by prying the seed loose from the position:
- Repetition: same prompt, same seed, five runs. All byte-identical means reproducible. If even one differs, the runtime drifts.
- Seeds: same prompt, five different seeds, one run each.
- Both are reported per model, never averaged across models.
- Comparison is byte-wise after normalising line endings. Token count per run is recorded alongside: same token count with different text is a different finding than a different token count.
Six models, ten runs each, on two machines: 120 cells. The task is an invoice with one correct answer — line discounts, an early-payment discount, a shipping threshold, VAT.
The result
| Model | 5× same seed | 5× different seeds |
|---|---|---|
ornith:35b | 5 identical | 5 identical |
qwen3.6:27b | 5 identical | 5 identical |
gemma4:26b | the first differs | 5 identical |
qwen3:8b | the first differs | 5 identical |
deepseek-r1:14b | the first differs | 5 identical |
qwq:32b | the first differs | 5 identical |
The right-hand column is the answer. Five different seeds produce exactly the same text on all six models — and it is the same text that repetitions two through five produce under seed 42. The seed value changes nothing. The only thing that differs is the first run, and it differs even though it carries the same seed as the four that follow it.
The token counts show it most plainly. gemma4:26b across its five repetitions: 503, 508, 508, 508, 508. qwen3:8b: 716, 723, 723, 723, 723. The first run is the shorter one — with identical numbers on both machines.
What the difference looks like
Verbatim from the stored outputs, gemma4:26b, the same point in the calculation. Up to here both versions are character-identical; from here they part. First run:
**2. Anwendung des Skontos (3%):**
854,00 EUR × (1 - 0,03) = 854,00 EUR × 0,97 = **828,38 EUR**Runs two through five, same seed:
**2. Anwendung des Skontos (3%):**
Der Skonto wird auf die Zwischensumme angewendet:
854,00 EUR × (1 - 0,03) = **828,38 EUR**Both calculate correctly, both reach the same intermediate figure, and both close with the same line: ERGEBNIS: 985,77 EUR. That holds for all four models whose first run differs. The difference is in the wording, not in the answer. Check only the final number and you will never see it; check for byte equality and you will write the model off as unreliable, when it did the same arithmetic twice.
Two machines, 108 cell pairs, not one difference
Comparing across two machines is the strongest evidence this series has, and here it comes out unusually clean. Both boxes carry an RTX 3090; beyond that they share little — Windows against a Linux container, different CPU, different everything below.
| Data set | Cell pairs | byte-identical |
|---|---|---|
| determinism, 6 models | 60 | 60 |
laguna-xs-2.1-stop, 16 groups | 48 | 48 |
It is not only the results that agree — the deviation agrees too. Where the first run steps out of line on one machine, it steps out of line on the other in exactly the same way, same text, same token count. For laguna, the same twelve of sixteen groups split the same way on both hosts: seed 42 alone, 43 and 44 together.
That is the actual finding. Nothing here is random. There are two paths, each fully deterministic in itself and reproducible across machines, and which one you get is decided by the position of the run.
The counter-test that split the question
That leaves the cause. The obvious explanation is prompt cache reuse: the first time the prompt is computed, afterwards it is already there, and the second path yields marginally different intermediate values. Obvious is not measured, so I measured it — with the prediction written down first:
- Series A: the measured prompt six times back to back, same seed.
- Series B: six times the sequence [foreign prompt, measured prompt]. The foreign prompt is there to overwrite the cache.
- Prediction if the cache is the cause: every run in series B is a first run, so all six must match the first-run version.
- Prediction otherwise: series B behaves like series A. If that happens, the explanation is refuted, and that is the result.
- Control: a model with no deviation in series A must show none in series B either. If it does, the setup is broken, not the finding.
Both happened — depending on the model:
| Model | Series A | Series B | Verdict |
|---|---|---|---|
gemma4:26b | the first differs | all six are follow-up | refuted |
qwen3:8b | the first differs | all six are follow-up | refuted |
deepseek-r1:14b | the first differs | all six are first-run | confirmed |
ornith:35b | no deviation | no deviation | control clean |
On two models the foreign prompt does not restore the first run. On deepseek-r1:14b it does, exactly as predicted — there, with a prompt in between, every run is a first run. So the cache explanation is neither refuted nor confirmed but too simple: whatever makes the first run take a different path behaves differently from model to model.
ornith:35b ran along as the control: six identical outputs in both series. So the setup does not manufacture the effect. The hash of those six runs is the same as the one from the determinism run two days earlier — under a different generation limit, in a different session.
One side finding clears the suspicion that all of this is an artefact of truncation: this run had the limit at 4000 tokens instead of 1200, and deepseek-r1:14b finished in all twelve cells. The first-run difference therefore shows up in a fully completed answer as well.
Two mistakes of my own
The first was a truncated comparison. In the determinism run, qwq:32b and deepseek-r1:14b sat at done_reason: length in all twenty cells — the generation limit was 1200 tokens and not one answer ever finished. What was compared there was a truncated opening. The run log said nothing about it; it only reported “STREUT”. For those two models the finding therefore covers the first 1200 tokens only. For the other four the answers are complete.
The second was a swapped model. I had noted that seed invariance was hardware-dependent: laguna giving the same output for all three seeds on one machine and not on the other, parting at byte 20. Re-checked with the same model on both sides, all 48 cell pairs are byte-identical. The split at byte 20 reappears precisely when you hold laguna-xs-2.1-stop against laguna-xs-2.1 — two variants, same size, different results. I compared two machines and in doing so compared two models. The finding was a comparison error, and the rule it broke is my own.
What this does not show
- The cause is unknown. What is measured is that position matters and the seed does not. Why the first run takes a different path is open — the one explanation I tested holds for one model out of three and fails for two.
- Everything here was measured at
temperature 0. It says nothing about higher temperatures, where the seed is supposed to matter and probably does. - One task family, one runtime (Ollama 0.32.5), one class of GPU. Two machines are two machines.
- 16 of the 176 groups contain at least one truncated cell. They are included in the counts, and there the comparison is incomplete.
- The counter-test used one foreign prompt. That some other arrangement does restore the first run is not ruled out.
The hardware, and the rules every measurement on this site follows, are on How I Measure.
What I take from it
First, for this series: “three seeds” was never seed variation. It was three repetitions, the first of which systematically steps out of line. The measured numbers stand — the justification, that they caught drift, does not. It has been corrected with dated notices in fourteen posts since 2 August.
Second, for anyone comparing local models: the first run is not a measurement like the others. Put two models or two builds head to head, run each once, and you may be comparing two first runs, each on its own detour. A warm-up run that you throw away costs a few seconds and removes an entire class of error.
Third, and this part reaches past language models: the finding had been lying in the data all along. It took no new instrument, only a different question — which of the three cells differs, instead of how many differ. The figure “only 65 of 176 groups are reproducible” sat in my notes for a month. It was correct, and it was misleading.