I gave a 120-billion-parameter mixture-of-experts model a second GPU. It got five times slower. Here is the measurement, and the more useful thing I found by accident.
The setup
Two RTX 3090s in one box. No NVLink — the cards talk to each other over PCIe 3.0 and nothing else. The model is gpt-oss-120b, MXFP4 quantised, far too large for a single 24 GB card. The serving engine keeps the routed experts in host RAM and streams them to the GPU on demand, so this workload lives and dies by bus bandwidth.
One card versus two
Same session, same prompts, both cards otherwise idle:
| 255 tokens | 799 tokens | |
|---|---|---|
| 1 GPU | 16.0 tok/s | 25.8 tok/s |
| 2 GPUs (tensor parallel) | 4.6 tok/s | 5.0 tok/s |
| −71 % | −81 % |
It worked, technically. Both cards filled up (about 21 GB each), the answers came back complete. Tensor parallelism splits every layer across both GPUs, which means an all-reduce on every forward pass. Over PCIe 3.0 with no NVLink, that traffic costs more than the second card contributes. Two cards are not “more GPU” for this. They are one GPU plus a tax.
The accident worth more than the experiment
Earlier the same day I measured the single-GPU case at 8.3 and 10.0 tok/s — roughly half of what you see above. Nothing about the software changed. What changed was which card the job landed on.
One of my two slots runs at x8, the other at x16. When only the x8 card was visible to the container, the job used it. When both were visible, the engine picked the x16 card. Same model, same code, 2.6× difference.
For anything that streams weights over the bus instead of keeping them resident, the slot is not a detail. If you have one fast slot and one slow one, know which is which.
Two ways I fooled myself
The engine failed to build CUDA collectives — and it broke the single-GPU case too. Missing NCCL surfaced as a linker error, but only after I made the second card visible. The single-GPU run had worked purely because the other card was hidden. Changing GPU visibility changed the code path, not just the capacity.
My readiness probe lied. I polled /v1/models, which answers with 200 long before the weights are loaded. The server looked ready after 13 seconds, and I recorded three “measurements” that were all HTTP 503. The honest probe is a real completion request, with 503 meaning “not yet”. Actual load time for this model: about 10 minutes.
What I take away
The interesting result is not that tensor parallelism is slow without NVLink — that is expected. It is how much of my day went into two measurement errors that each looked exactly like a real finding. A number you did not check the provenance of is not a measurement. It is a rumour with a decimal point.