Local LLM Archives - Tobias Liauw

Two panels contrasting a summary table with the raw data. Left, what the table said: model A passed 5 of 5, model B only 2 of 5, a clear ranking. Right, what the raw data said: model B produced valid, animated code in 10 of 10 runs, and the entire gap rested on a word search in the source — naming a quantity F_b instead of buoyancy caused a failure. A panel below records that the 198 outputs backing an earlier article lived on a single machine that was wiped, with no download link ever published.

Aug 15, 2026

If You Publish Benchmarks, Keep the Artifacts

In July I published an article about determinism in local models. It contains this sentence: “All 198 outputs are available in full, so that you can…

Two-panel board on a very dark ground. Left, the first result: 73 percent against 99, a 26-point collapse, 0 of 45 points on the logic puzzle in 2 of 4 prompt variants, while the other three tasks stayed at 100 percent. Right, the corrected result: the abliterated build scores 336 of 336 points, the base model 334 of 336 (99.4 percent); of 1,320 re-scored cells exactly six changed, five reading ERGEBNES and one ERGEBNET instead of ERGEBNIS, all six containing the right answer. A strip below shows format compliance at 42 of 48 cells, broken down as bare 9 of 12, steps 9 of 12, role 12 of 12, example 12 of 12. Tags at the bottom read 48 cells against 48 cells, second machine digit for digit, median 535 to 564 tokens, median 5.6 to 5.8 seconds.

Aug 2, 2026

What abliteration costs — and what it doesn’t

An abliterated model scored exactly zero out of 45 on the logic puzzle. "Abliteration destroys reasoning" would have made a good headline. It just was not…

Split panel. Left, across both machines: 90 of 99 outputs byte-identical, 96 of 99 cells scored the same, 198 requests with zero errors. Right, the exception qwq:32b: it needs 24.8 GB on a 24 GB card, runs 6.6 percent on the CPU on the workstation and 3.5 percent on the headless container, and accounts for all 9 divergent outputs. A strip below notes that the first comparison said 2 of 99 because the Windows host writes CRLF line endings and the Linux host writes LF. Along the bottom, the differences between the two hosts: Windows 11 against Ubuntu, 128 GB against 24 GB of RAM, NVIDIA driver 610.88 against 580.126.09, an EVGA board against a Zotac, 11 models and 3 tasks.

Aug 1, 2026

Same Prompt, Two Machines: 90 of 99 Outputs Were Byte-Identical

Eleven local models, three real tasks, two RTX 3090 hosts, 198 requests, zero errors. 90 of 99 outputs came back byte-identical across machines — and the…

Grouped bars compare points per task before and after the stop token was added, in percent: rechnung falls from 88 to 75, extrakt rises from 0 to 100, raetsel from 30 to 39. A side panel carries the totals across all 48 cells: 117 of 336 points before against 215 of 336 after, cells at the token ceiling 48 of 48 against 3 of 48, format compliance 31 against 83 percent, 192,000 against 45,775 generated tokens, 1594 against 326 seconds of compute time, and the plan task unchanged at 88 percent.

Aug 1, 2026

The model that couldn’t stop

One model sits at the bottom of the table with 35 percent. The number was measured honestly — and is still wrong, in both directions at…

Bar chart of generation throughput in tokens per second at 64k context for three models under two conditions, context merely reserved against context filled to 84 percent: qwen3-vl-abl:30b 149.2 against 39.1, glm-4.7-flash 145.0 against 58.4, ornith:35b 108.3 against 85.6. A side card records that all three ran with 0 percent of the model on the CPU, that in the paired run only qwq:32b offloaded anything, that an empty 128k KV cache costs about 1.5 GB, and that a full cache has to be read for every generated token.

Jul 31, 2026

Reserving 128k Context Costs Nothing. Filling It Costs Two Thirds.

Two RTX 3090s, eleven models with identical digests, 528 requests and no errors. Reserving a 128k KV cache is nearly free; filling it costs 10 to…

Two panels on a very dark background. Left, headed First reported: every model gains 3–11 percent, reported for four models, old mean against new median. Right, headed After recomputing: −2.6 percent to +3.2 percent, eleven of twelve models under 3 percent, the gains vanished entirely. A strip below covers laguna-xs-2.1 as broken: the token cap hit on 30 of 30 reasoning requests, at exactly 900 and exactly 3,000 tokens, where 0.31.2 used 215 and 897 tokens and never truncated, because the stop token is not honoured. Footer tags read 12 models across 9 architectures, 11 tasks with 3 runs each, the same RTX 3090 on driver 610.88, and qwq at +78.7 percent marked as a context effect rather than a version effect.

Jul 31, 2026

Upgrading Ollama Made Nothing Faster — But It Broke One Model

Ollama 0.32.5 was out, my workstation was still on 0.31.2, and I had just finished benchmarking 26 models on it. So I had a rare opportunity:…

Grouped bar chart comparing three GPU sensors under two loads on the same RTX 3090 at 350 W. Core reaches 78.6 °C under LLM inference versus 77.7 °C in 3DMark Steel Nomad. Hot spot reaches 93.7 versus 91.8 °C. Memory junction reaches 104.0 versus 92.0 °C, twelve degrees hotter under inference. A side panel gives 18 °C of headroom to the 110 °C throttle in the graphics test but only 6 °C under LLM load, and thermal throttling in 2 of 7,432 samples, 0.03 percent.

Jul 30, 2026

I Benchmarked 26 Local LLMs on One RTX 3090 — Here Is What Actually Held Up

I have an RTX 3090 sitting in a Windows workstation that is about to be torn down and rebuilt as a Proxmox AI host. Before that…

A dark two-panel comparison. Left, the rule as originally written: three seeds at temperature 0, so that one lucky answer cannot carry a model, on the assumption that the seed varies the output. Right, the re-measurement: the seed has no effect, exactly one run per group differs, and it is always the first, in 111 out of 111 cases, so there is no variation of the seed. A strip below states that five different seeds produce the same text as five repetitions of one seed. Tags at the bottom read temperature 0, discarded warm-up run, medians not means, and measurements unaffected.

Jul 26, 2026

How I Measure: The Ground Rules Before Any Numbers

Two hosts that share a graphics card and almost nothing else, scoring that only counts what a script can verify, and the rules every number in…

Two fields side by side on a dark canvas. Left, prefill: the whole prompt is read in parallel, limited by raw compute, usually running at thousands of tokens per second. Right, generation: one token at a time in sequence, where a 1,000-token answer means 1,000 full passes through the network, limited by memory bandwidth rather than compute. A strip beneath both fields shows the VRAM cliff: if the model does not fit, layers run on the CPU and move data over PCIe instead of at VRAM bandwidth, which is not a gentle degradation. Terms listed below: token, KV cache, prefill, decode, offloading. Closing line: two phases, two bottlenecks, one misleading number.

Jul 25, 2026

Starting a Series on Local Models — First, the Vocabulary

The opening piece of a series in which I test what local language models actually do on hardware you can buy. This first part has no…

Concept diagram on a very dark ground. At the centre a purple-outlined circle reads VENICE, Privacy-First AI. On the left a card labelled "You (User)" lists encrypted prompts, zero logs, zero storage and notes that privacy sits in the architecture. On the right a card labelled "GPU Providers" lists decentralised compute, open-source models and the names Llama, Mistral and Gemma. Dashed lines connect both to the centre, labelled "encrypted req" and "inference". Below, a card on VVV token staking carries three badges — stake for access, earn from GPUs, no monthly fees — with the note that tokens are staked rather than spent and can be unstaked at any time.

May 22, 2026

Venice.ai: Privacy-First AI With Token Staking Instead of Subscriptions

Venice.ai runs open-source models with zero-knowledge privacy. No prompt logging, no training on your data. The VVV token staking model gives you permanent AI access without…