Two years ago I wrote about a NUC running 8B models on CPU and called that the whole story. The fleet has grown since then. Today it’s three machines with very different jobs: a dual-GPU server handling six parallel 128k sessions, a 121 GB unified-memory DGX Spark that owns everything over 128k tokens, and a laptop with a ten-year-old GPU that is, surprisingly, the fastest single stream in the whole setup. This is how each one got built, and how a ~200-line Python router keeps every client working while the backend quietly switched from ollama to llama.cpp.
All numbers in this post were measured, not estimated. Where I had to estimate, I say so.
The fleet at a glance
Three machines, three jobs, one Tailscale mesh. No public ports, no API keys — the mesh is the trust boundary, the servers are unauthenticated on purpose.
| Machine | Model | Slots | Measured |
|---|---|---|---|
| Atlas — 2× RTX 3090, LXC on Proxmox | Qwen 27B abliterated (Q4_K) | 6 × 128k context | 105–109 tok/s aggregate at 6 parallel |
| GB10 — DGX Spark, 121 GB unified | Qwen 27B-class MoE, abliterated IQ3_S (84 GB) | 4 × 262k context | 26.6 tok/s generation, ~970 tok/s prefill |
| Laptop — RTX 2080 Ti, 11 GB | 9B class (Q4_K_M) | single stream | 54.7 tok/s — the fastest single stream on the fleet |
Atlas: the workhorse, rebuilt around slots
Atlas is the main infrastructure host — Proxmox, and on it an LXC container that carries the GPU pair (two RTX 3090s on passthrough). For a long time inference ran through ollama, which is a fine tool, but ollama serializes: one request at a time. Under fleet load — and this fleet is 25+ hosts running agents — the time-to-first-token under contention hit 243 seconds. That’s not a model problem, that’s a scheduling problem.
The fix was a llama.cpp build with a flag most people never touch: -np 6 (six parallel slots) plus -kvu --kv-unified-per-slot 131072 — a unified KV cache carved into six equal 128k windows. The model (a 27B Q4_K, ~17 GB) sits once in VRAM; each session gets its own 128k of KV. Measured results on the same harness that gave old-ollama 48 tok/s serial:
| Concurrency | Per session | Aggregate | First token |
|---|---|---|---|
| 1 | 42.6 tok/s | 42.6 | 0.32 s |
| 3 | 31.8 tok/s | 87–91 | 1.0 s |
| 6 | 19.2 tok/s | 105–109 | 2.5 s |
Three concurrent sessions run faster in aggregate than the best single stream. That’s the whole argument for slots.
Why 6 × 128k and not 3 × 256k
It looks like you’d prefer fewer, bigger windows. The math says otherwise, and it comes from three measurements I made against my own prompt history:
- Identical KV cost. 6 × 128k and 3 × 256k both reserve 786,432 KV cells — ~20 GB at q4_0 quantization (~27 KiB/token on this model). Bigger windows buy nothing on memory.
- The workload doesn’t want it. 3,408 real prompts from a week of fleet traffic: only 5.9% exceeded 128k tokens. The median prompt was 245 tokens. The 250k edge case would need minutes of prefill on a 3090 before the first token appears.
- Headroom for the rest. 8 × 128k (27 GB KV) wouldn’t fit with the model, embeddings and OS in 48 GB across two cards. And the long cases have a better home — see the GB10.
GB10: the long-context machine
The GB10 (DGX Spark) is a 121 GB unified-memory arm board — essentially a workstation where the CPU and GPU share one big memory pool. That property is exactly what long-context inference needs, because context length is a memory problem, not a compute problem.
The model on it is a 27B-class mixture-of-experts (512 experts, ~10 active per token) in an 84 GB quantization — it fits because the experts stream through the 121 GB pool. Two architectural details make this machine unusual:
- The KV cache is tiny. The hybrid attention design keeps full-attention layers to a small fraction of the stack (12 of 48). Four 262k slots with a unified KV pool cost very little memory — the server’s entire footprint, model included, is 61.8 GB.
- It’s bound, not exposed. The API listens on the Tailnet address and localhost only, as a systemd user service with linger — no sudo, no public surface. Measured: ~970 tok/s prefill, 26.6 tok/s generation. Slower per token than the 3090s, but it’s the only place a 200k-token prompt can live at all.
Routing rule in one sentence: anything over 128k tokens goes to the GB10, everything else stays on Atlas or the laptop.
The laptop: accidental champion
A 10-year-old RTX 2080 Ti with 11 GB of VRAM. I put a 9B-class model on it (6.5 GB, Q4_K_M) and it pushes 54.7 tok/s — faster per session than any single stream on the dual-3090 box, because it’s doing one job at a time with zero contention. The lesson that repeats itself in this fleet: a small model with a whole GPU beats a big model sharing one.
Practical split that came out of that: short, interactive, single-stream work (pair programming, quick reviews) runs locally at 55 tok/s. Anything that needs the 27B’s quality, or any parallelism, goes to the fleet. The laptop isn’t a fallback — it’s the right tool for its niche.
The router: keep the old port, swap the brain
Here’s the part most setups get wrong. You’ve now got three backends (llama.cpp, ollama, the GB10), and 25+ clients across the mesh — agents, CLI tools, a paperclip-style agent platform — all hardwired to one URL: atlas:11434. Reconfiguring every client is a project; keeping the port and routing by content is an afternoon.
So Atlas runs a small router — Python standard library only, ~200 lines, one systemd unit — on the old port 11434. The rules:
- Model
qwen3.8:27b(or:latest) → GPU llama.cpp on 127.0.0.1:8080, which speaks the OpenAI API. Everything else (embeddings, the small utility models) → real ollama, now bound internally on 127.0.0.1:11436./api/tagsand similar pass straight through. - API translation where needed. Old clients speak ollama-native (
/api/generate,/api/chat); the new backend speaks OpenAI (/v1/chat/completions). The router translates both directions, streaming included — think/format/tool-call payloads and all. New clients can hit the OpenAI routes directly on the same port. - A quirk worth knowing: Claude Code splits its system prompt across multiple blocks and can drop a
role:"system"message mid-conversation. Qwen’s chat template rejects that with a hard 500 (“System message must be at the beginning”). The router folds all of it into one system block on the Anthropic route before forwarding. One bug class, fixed in one place, for every client.
After the cutover: 43 models still visible on the tags endpoint, embeddings unchanged (1024 dims), chat working over LAN and Tailnet, zero failures across 80+ requests. The clients never knew. That’s the test that matters for this kind of change — the old port keeps its contract, the new backend gets the traffic.
One operational detail that bit us and is worth stealing: when llama.cpp restarts, it frees VRAM — and ollama, which is always hungry, grabs it back in the restart gap, so llama.cpp comes back OOM. The systemd unit now has an ExecStartPre that unloads the other models before the server reserves memory, and the qwen model in ollama is pinned to CPU as a last-resort copy. A four-minute outage taught me that.
What I’d do differently
- Measure the prompt distribution first. The 6 × 128k design only made sense after I pulled a week of real prompts and saw the 5.9%/245-token-median. Guessing at context sizing is how you end up with either wasted VRAM or 200-second first tokens.
- Slots from day one, not after the 243-second TTFT. The migration from ollama to llama.cpp felt like a big move. It was really a scheduling fix with a model in front of it.
- Give the long tail a dedicated home before it becomes a problem. The GB10 was a “what is this board” purchase; the hybrid-attention model’s tiny KV cache is what turned it into the long-context tier of the fleet. The hardware was the easy part.
- Keep the router dumb. Standard library, one file, two upstreams, rules by model name. The moment it needs a database, you’ve built a service, not a router.
The recurring theme: three machines that look unrelated actually solve one problem each — parallelism (Atlas), memory (GB10), and contention-free single streams (laptop) — and a thin router that lets the fleet treat all of it as one endpoint. No cloud, no per-token bills, no public ports. And the whole thing runs on hardware that was a one-time cost.