Four times past the trained window: one million tokens still finds the needle, in 14 minutes

Two nights in a row I pushed the same model past its trained window: 524,288 and 1,048,576 context tokens on two GeForce RTX 3090s, won with rope stretching (YaRN, scale 2 and 4), run through the production engine, with a restart after every configuration. The question was short: past the trained window, does the model still find a needle in the prompt — and what does that cost in time? The answer in one sentence: It finds it more reliably than I expected — 24 of 24 probes across twelve cells, three KV tiers, two window sizes — but the time does not grow linearly with the context, it grows superlinearly: 5.8 minutes for a full pass at 512k, 13.7 minutes at one million, and both k8v4 double-session cells at 512k collapsed along the way. This is part two of the series on long context on the same machine; part one measures what the KV cache costs at 262,144 tokens.

If you want the ground work: part one — KV memory and the wait at a full window, plus how the fleet is built, the GPU patch (active in every measurement here) and the quant ladder of the same model.

What “stretched” means technically

The model — qwen3.8-flash-next-abliterated, a mixture-of-experts build in IQ3_S — was trained with a rotary window of 262,144 tokens. YaRN shifts the rotation frequencies of the RoPE keys so the same keys map a multiple of that distance without leaving the training regime; the engine gets this as a parameter, not as new weights:

• --rope-scaling yarn, --rope-scale 2, --yarn-orig-ctx 262144 → the window reports 524,288 tokens.
• --rope-scale 4 with the same origin window → the window reports 1,048,576 tokens, exactly four times the trained range.
• KV cache grows linearly with the window regardless: int8 stays at 22,176 bytes per token, so 5.41 GiB at 262,144, 10.83 GiB at 524,288 and 21.66 GiB at one million. q4_0 (4-bit keys and values) uses 12,096 bytes per token: 2.95 / 5.91 / 11.81 GiB. k8v4 (8-bit keys, 4-bit values) sits between them: 4.18 / 8.37 / 16.73 GiB. Computed from the format sizes, not measured.

Relevant for how to read this: YaRN is a documented, mathematically clean method — but whether a specific model still produces usable answers beyond its training window is not part of that documentation. That gap is exactly what this post fills: not the theory, but twelve cells with a needle and a stopwatch.

How it was tested

Measured in phase a2 of the sweetspot matrix: 20 cells, across two nights from 29 October 18:13 to 30 October 04:08, each after an engine restart with the new configuration. Rotated were three KV tiers (int8, k8v4, q4_0) × two KV residency settings (20 GiB and 64 GiB) × two window sizes (512k with scale 2, 1M with scale 4), one session each — those are the twelve cells with a needle test — plus eight cells with two concurrent sessions for parallel behaviour.

Two instruments, kept apart:

• The timing probe sends a fresh 262,144 token prompt and measures first token, prefill rate and decode rate. It is the same size in all three windows, which makes it the comparison point: what does a stretched window cost when the work stays identical?
• The needle test builds a prompt that nearly fills the window, hides a code word with a number at half and at top depth, and asks for it back. Thinking is off, temperature 0, at most 32 answer tokens — so the answer time is essentially the prefill time of the entire prompt. Every cell got two probes.

Two honesty notes up front. First: at the top edge of the window the server refused the prompt several times (HTTP 400) and the checker then shrank it — that happened for 5 of 12 probes at 512k and 7 of 12 probes at one million. The prompts therefore landed between 445,471 and 524,123 tokens (512k) and between 890,788 and 1,048,479 tokens (1M). Not every probe was measured at the absolute stop; most were just below it. Second: in stretched context I tested two depths (50 and 90 percent), in native 262k context the preparatory run used three. The hit rates of the two posts are therefore not row-by-row comparable.

Finding 1: Past the trained window the model still finds the needle — in all twelve cells

Twelve cells, two probes each, twenty-four hits. No miss at half depth, no miss at top depth, neither at scale 2 nor at scale 4, neither with int8 nor k8v4 nor q4_0. That is a different result than at native 262,144: there, a preparatory run missed the probe sitting on the very last edge (prompt 262,133 out of 262,144 tokens). So the edge of the native window is less reliable than the interior of a window stretched four times.

Two caveats that do not shrink the result but place it. The needle is a retrieval task: a code word with a number sits in the text and must come back. It tests attention over long distance, not reasoning across paragraphs, not arithmetic, not contradiction detection. And: at one million context the 50 percent probe is itself already 524,000 tokens deep — also outside the trained range; it was found anyway. What I take from this: for retrieval in long documents the stretched window is usable. For quality beyond retrieval I have no evidence.

Finding 2: The timing probe barely notices the stretching — the full pass notices it clearly

The first column is the good news: for a 262,144 token prompt the engine needs 131.8 seconds to the first token inside a 512k window, 138.1 seconds inside the million-token window — and 139.5 in the native window. The stretching itself costs measurably nothing; the compute for a prompt of that size is identical no matter how much further the window stays open above it.

The columns on the right are the bad news, and they are the more interesting ones. Fill the window for real — 485,000 instead of 262,000 prompt tokens — and the effective rate drops from 2,053 to 1,454 tokens per second; at one million prompt tokens it drops to 1,210. The last 500,000 tokens of a prompt cost noticeably more time per token than the first 262,000. Whether that comes from quadratically growing attention, from KV paging, or from both, I did not measure — measured is only the consequence: whoever uses the window does not pay proportionally, but superlinearly.

The decode cadence is untouched by all of this: median 49.5 tokens per second at 512k, 49.6 at one million — practically the number from part one. What changes is exclusively the time until the first token. With long context, prefill decides, not output.

Finding 3: A full pass costs 5.8 to 13.7 minutes — and the KV tier is indifferent

The median over all six 512k cells is 348 seconds (5.8 minutes) for a single fully answered request, the median over all six million-token cells 819 seconds (13.7 minutes). For scale: 262,144 native tokens cost 153 to 240 seconds in the same measurement series — so the step from 512k to one million is not another increment, it is a doubling.

What stands out is how flat the curve stays across KV tiers: the tier medians read 324.6 / 356.8 / 383.6 seconds at 512k and 819.0 / 871.8 / 783.9 seconds at one million. Across three tiers whose memory needs differ by a factor of 3.7 (5.91 against 21.66 GiB at a million), that is an 18 percent spread at 512k and 11 percent at one million. The tier decides memory, not answer time. Whoever needs room at one million takes q4_0 and loses nothing measurable.

Finding 4: Two sessions at 512k — and the puzzle of the two k8v4 cells

Five of the eight double cells behave unremarkably: two parallel requests cost the same lead time as a single one and together deliver 71 to 77 tokens per second — the pattern I know from part one. Two cells stick out brutally: both k8v4 cells at 512k, at 8.7 and 12.8 tokens per second with parallel numbers of 9.7 and 16.5 — that is standstill, not sharing.

I have no explanation for it and I claim none. The pattern is loud enough to write down: two of two k8v4 double cells collapsed, two of two int8 and two of two q4_0 double cells did not, same night, same machine, same window. Plausible directions would be the mixed dequantisation path (8-bit keys, 4-bit values) under parallelism, or memory pressure, since k8v4 with 8.37 GiB per session sits between the tiers — but I tested neither. Until the counter-test I do not schedule k8v4 for two sessions in the big window. At one million I only ran the double cells with q4_0; nothing broke there.

Finding 5: The GB10 holds 262,144 tokens, but with nearly triple the lead time

On the GB10 with its single unified memory (121 GiB, of which the model occupies 51 GiB), 262,144 tokens run cleanly in both tiers with a hit in all three tested depths — but prefill takes 379 to 405 seconds instead of 139, the full pass 472 to 502 seconds instead of 139 to 261 in the native 3090 comparison cells (two depths tested there, three here). Extrapolated to one million tokens, where int8 KV needs rise to 21.66 GiB and the 3090 pair already needs 13.7 minutes per full pass, the GB10 would sit at roughly 35 to 40 minutes per request. That is arithmetic, not measurement — and the reason long context stays on the two cards in my setup while the GB10 has taken over the embedding and retrieval models.

What I did not measure

• No quality axis beyond the needle. No perplexity, no summarisation or arithmetic task, no contradiction test across two halves of a document. Twelve cells of a retrieval task are a strong hint of usable long-range attention, not a quality verdict.
• Only two depths in stretched window (50 and 90 percent) instead of three as in the native run-up. The very shallow and very deep edges of the stretched window are untested.
• One run per cell. The medians of the speed columns at 512k and 1M rest on one measurement per configuration each, the answer times on two probes. The tier ranking (finding 3) is noise with that sample size, the order of magnitude of the answer times is not.
• No YaRN scale beyond 4, no dynamic or LongRoPE variants, no continuous curve at constant quality: I measured two points, not the line.
• Not measured: why k8v4 breaks down under parallelism — and whether it reproduces.
• Production traffic kept running, at night mainly router and embeddings on the GB10. The timing probes of the stretched cells sit between 1,557 and 2,092 tokens per second — part of that spread is shop floor noise.

What I do with it

Four sentences that remain from these two nights. First: 524,288 tokens are operationally ready for me — one request that fills the window takes 5.8 minutes and finds what it is supposed to find; that is a lookup job, not an interactive one. Second: one million tokens is doable and noticeably at the edge: 13.7 minutes per request, single-session, and the prompt lands practically against the stop (largest request processed: 1,048,479 of 1,048,576 tokens). From now on I plan windows with 5 to 10 percent headroom at the top, because the stop has edge problems — at native context a probe was missed there, in stretched context 12 of 24 probes had to be shrunk after an HTTP refusal. Third: q4_0 is the sensible tier for stretched context: 11.81 GiB instead of 21.66 at a million, and measurably level on answer time. Fourth: stretching is not a free window. As long as I need 131,072 tokens I take the native window with 66 seconds of lead time; the stretch does not cost the window, it costs the superlinearity behind it.

Open for the next night: the counter-test for k8v4 under parallelism, a quality task beyond the needle at 512k, and the million-token cell on the GB10. Once measured, I will write it up.

Artefacts

The measurements of this phase as open files: all figures as JSON (medians per cell, rope scale, individual needle results with prompt sizes and remarks, origin timestamps) and the cells as CSV (phase, platform, rope scale, KV tier, residency, window, sessions, prompt size, first token, prefill, decode, parallel throughput, needle hit rate, answer time). Part one of the series is here, including the full cell matrix of the KV axis.

The four figures at full size (1200 × 630): A million tokens takes one work step · The stretching itself costs nothing · Everything is still found — time is the price · What I can actually do with 1 M tokens.