A single consumer GPU hits a wall in two places: the model is bigger than its VRAM, or the throughput is not enough for what you are doing. The “buy another card” answer works for the first and is usually wrong for the second. This page is the read-first guide to which problem you actually have, which runner does what, and which interconnect makes the difference between a multi-GPU box that works and one that crawls.
Verified September 2026 against Ollama FAQ and GPU docs, the llama.cpp server README and build.md, the vLLM engine-args and expert-parallel pages, the Qwen3-235B-A22B model card and HF API, the Ollama library qwen3 tags, the NVIDIA H100 / RTX 4090 / RTX 5090 spec pages, and Apple’s Mac Studio spec page.
What One Card Cannot Do
The model-is-bigger-than-the-card symptom is the easiest to spot. qwen3:32b is 20 GB at the default quantisation (Ollama qwen3 tags); it just fits on a 24 GB card and is uncomfortable on a 16 GB one. qwen3:235b-a22b is 142 GB (Ollama qwen3 tags) - not a single-card build under any consumer-grade quantisation, and the question this page is mostly written to answer.
The throughput symptom is quieter. A 24 GB card running qwen3:30b-a3b at 19 GB (Ollama qwen3 tags) can serve one user at a useful speed. Three users at once will not get one-third of the speed each - they will queue, and the user who hits “send” last will wait. A second card does not make one model faster; it makes two users at full speed possible.
Note that “more users” is a third problem with a different answer - it is mostly a data parallel problem (multiple copies of the model, one per request) until you run out of cards. Splitting one model across many cards is rarely the right way to add seats.
Two Ways to Split One Model Across Many Cards
There are two technical patterns for putting one model on more than one GPU, and the runners expose them under different names.
Layer split (pipeline parallelism). Each card holds a few transformer layers. A token enters card 0, hops to card 1 when its layers are done, and so on. The big plus is that the cards never need to talk to each other in the middle of a layer, so PCIe-only boxes work fine. The big minus is that the slowest card sets the pace. llama.cpp calls this --split-mode layer, which is also the default (llama.cpp server README).
Tensor parallelism. The weights and KV cache of each layer are split across cards, and the cards must exchange partial sums every step. This is what vLLM calls --tensor-parallel-size (default 1, per the vLLM engine-args page). llama.cpp exposes the same idea as --split-mode row and --split-mode tensor, with --tensor-split 3,1 style fractions to bias which card gets how much. This pattern is fast on fast interconnects and slow on slow ones.
A third, separate pattern - data parallelism - is what you actually want for “more users”. You run N independent copies of the same model on N cards and route requests between them. llama.cpp’s --parallel N does something similar inside one server, vLLM calls it --data-parallel-size. This page is not about that pattern; this page is about one model, too big for one card.
| Pattern | Runner flag | When to use it |
|---|---|---|
| Pipeline (layer split) | llama.cpp --split-mode layer (default) | Two or more cards, no fast interconnect, model too big for one card |
| Tensor parallel | vLLM --tensor-parallel-size N; llama.cpp --split-mode tensor / --tensor-split | Cards with NVLink or NVSwitch; small to mid model size that still doesn’t fit |
| Data parallel | vLLM --data-parallel-size N; llama.cpp --parallel N | Same model, more users - this is not “one model on many cards” |
What the Runners Actually Do
Ollama’s behaviour is the simplest: if the model fits on any single GPU, Ollama loads it on that one GPU; if it does not, Ollama spreads it across every visible GPU (Ollama FAQ). To force a subset you set CUDA_VISIBLE_DEVICES, ROCR_VISIBLE_DEVICES, or GGML_VK_VISIBLE_DEVICES to a numeric list before launch (Ollama GPU docs). This is layer-split behaviour in practice, and it works because Ollama is built on llama.cpp underneath.
llama.cpp is the most configurable. The full list per the server README is --split-mode none|layer|row|tensor, --tensor-split as fractions, --rpc to point at remote llama.cpp RPC servers, --main-gpu to pick where KV cache lives, --n-gpu-layers (or --ngl) to push some layers to GPU and keep the rest on CPU, and --parallel for concurrent slots. The build-time controls are listed in build.md: GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 lets a multi-GPU box swap to system RAM when VRAM is exhausted, GGML_CUDA_P2P enables peer-to-peer access between cards so data skips the CPU, and CUDA_SCALE_LAUNCH_QUEUES=4x is documented as a pipeline-parallelism throughput win.
vLLM is the production-oriented choice and assumes you know what you want. --tensor-parallel-size N plus --pipeline-parallel-size M splits one model across NxM GPUs. The vLLM parallelism-and-scaling page is explicit about the hidden catch: on boxes without NVLink (it calls out L40S by name), pipeline parallelism is faster than tensor parallelism because tensor parallelism carries higher communication overhead without an NVLink fabric (vLLM parallelism and scaling).
The Hidden Cost Is the Interconnect
Two PCIe Gen5 cards at 128 GB/s each can talk to the CPU at 128 GB/s and to each other at maybe 64 GB/s in practice. Two NVLink SXM cards can talk to each other at 900 GB/s (NVIDIA H100 page). That is not a 2x or 4x difference; it is roughly an order of magnitude, and the all-reduce traffic inside a tensor-parallel layer walks that gap every token.
Consumer cards do not have NVLink. The RTX 4090 spec sheet says “NVIDIA NVLink (SLI-Ready) - No” (NVIDIA RTX 4090 spec). The RTX 5090 spec sheet says the same (NVIDIA RTX 5090 spec). The newest consumer H100 sibling, the H100 NVL, ships an NVLink bridge at 600 GB/s (NVIDIA H100 page) but the bridge connects two H100 NVL cards only; you cannot bridge a 4090 to anything.
This is why the cheapest “two cards” path (two used RTX 4090s on a desktop motherboard) and the most expensive (a four-card H100 SXM node) are not even on the same problem. The two-4090 box is fine for layer-splitting a 32B-class model at modest speed; it is the wrong tool for tensor parallelism on a 70B. The H100 box is the right tool for tensor parallelism on a 70B at full speed. Picking the wrong one is the most common multi-GPU mistake.
| Configuration | Combined VRAM | Interconnect | Best workload |
|---|---|---|---|
| 2 x RTX 4090 (used) | 48 GB | PCIe only, no NVLink | Layer-split qwen3:32b; light tensor-parallel at small batch |
| 2 x RTX 5090 | 64 GB | PCIe only, no NVLink | Layer-split a 70B at Q4; tensor-parallel viable but not fast |
| Mac Studio M5 Ultra | up to 512 GB unified | 1.2 TB/s memory bandwidth | Any single model up to ~400 GB; one user at a time (Apple Mac Studio) |
| H100 NVL pair + bridge | 188 GB | NVLink 600 GB/s | Tensor-parallel on 70B-class at full speed |
| 4 x H100 SXM | 320 GB | NVLink 900 GB/s | Tensor-parallel on 235B-class; multi-user serving |
Unified memory is the awkward middle case. Apple’s M5 Ultra sits at 1.2 TB/s memory bandwidth (Apple Mac Studio), which is faster than PCIe and slower than H100 NVLink, and the 512 GB ceiling means models that would not fit anywhere else will run there. The catch is that Apple Silicon inference is bound by llama.cpp’s Metal backend, not the CUDA path, so fewer quants and slower upstream tuning for the newest models.
Stay Single-Card If You Can
Two RTX 4090s cost roughly the same as one H100, deliver roughly the same total VRAM, and do most workloads slower than a single H100 would. The reason to add a second card is not “I want more VRAM” - it is “I need this exact model and it does not fit on one card, or I need this exact throughput and one card cannot deliver it.” A weaker quantisation is often the cheaper fix: a 32B at Q4_K_M will sit on a 24 GB card where the same model at Q8_0 will not (Ollama qwen3 tags).
The other reason to add cards is seats. A small team of three to five using one box at once usually wants data parallelism (one model per user), not tensor parallelism (one model, many cards). For a workload like that, the local AI hub and the no-GPU entry point cover the rest of the planning; this page is the wrong place to start if your goal is shared access rather than bigger models.
Bottom Line
One card stops being enough in two distinct ways: the model does not fit, or the throughput does not satisfy you. For the first, layer-split (--split-mode layer in llama.cpp, Ollama’s automatic spread) is the cheap-and-works answer; tensor parallel (vLLM --tensor-parallel-size, llama.cpp --split-mode tensor) is the fast answer when the cards have NVLink, the wrong answer when they do not. For the second, the answer is usually more cards running independent copies, not the same model split wider. The interconnect bandwidth - not the VRAM total - is what separates a multi-GPU box that pays for itself from one that just pays for more parts.
The used GPU hub is the right next stop if you are pricing a build; the cost crossover page is the right next stop if the math on adding cards is what you actually need to settle.