Local AI by VRAM: Which Models Fit Your GPU - Best GPUs for Local AI - How Much VRAM Does a Local LLM Actually Need
Tooling: Best Local LLM Runners - Which Quantization Format - What Quantization Costs You
Every local AI setup asks the same question after the model loads: is this fast enough? The answer is not the model, it is the hardware. Local LLM inference speed is dominated by how fast the chip can move weights from memory into the compute units, and once you know that, the numbers fall out.
Updated September 2026. Refreshed as new consumer GPUs land and runners ship kernel updates.
Why Memory Bandwidth Is the Bottleneck
Autoregressive generation is a serial loop: feed in one token, read the next, repeat. At every step the entire model has to stream through memory. The arithmetic is small relative to the data movement, so wall-clock time per token is roughly weights_in_bytes / memory_bandwidth_in_bytes_per_second plus a constant overhead for the attention and KV-cache work. That is why the same model is faster on an RTX 5090 (1,792 GB/s) than on an RTX 4090 (1,008 GB/s), even though the 5090 has only about 30% more shader and tensor throughput (GeForce RTX 50 series specs, GeForce RTX 40 series specs). The bandwidth gap is the speed gap.
Apple Silicon tells the same story a different way. Mac Studio M5 Max delivers 460 GB/s and 614 GB/s (at the 40-core GPU tier); M5 Ultra pushes 1.2 TB/s (Apple Mac Studio tech specs). On identical models in MLX, that bandwidth delta is what moves tokens per second.
Tokens per Second at Common Tiers
The numbers below come from the official llama.cpp benchmark files. The single-batch column is what one user typing prompts into Ollama sees; the batched column is what a server delivering 32 simultaneous requests approaches.| Hardware | Model | Quant | Prompt t/s | Gen t/s | |---|---|---|---:|---:| | Mac Studio M2 Ultra | gpt-oss-20B | MXFP4 MoE | 2,713 | 130 | | Mac Studio M2 Ultra | gemma-3-4b-it-qat | Q4_0 | 2,924 | 134 | | Mac Studio M2 Ultra | Qwen2.5-Coder-7B | Q8_0 | 1,566 | 80 | | Mac Studio M2 Ultra | Qwen3-Coder-30B-A3B | Q8_0 | 2,453 | 79 | | Mac Studio M2 Ultra | GLM-4.7-Flash (30B-A3B) | Q8_0 | 1,629 | 60 | | Mac Studio M2 Ultra | gpt-oss-120B | MXFP4 MoE | 1,649 | 86 | | DGX Spark (GB10, 128 GB) | gpt-oss-20B | MXFP4 MoE | 4,506 | 83 | | DGX Spark (GB10, 128 GB) | gemma-3-4b-it-qat | Q4_0 | 5,949 | 81 | | DGX Spark (GB10, 128 GB) | Qwen3-Coder-30B-A3B | Q8_0 | 2,987 | 61 | | DGX Spark (GB10, 128 GB) | gpt-oss-120B | MXFP4 MoE | 2,444 | 59 |
Source: llama.cpp Mac M2 Ultra bench, llama.cpp DGX Spark bench.
Two things stand out. First, the small dense model (gemma-3-4b at Q4_0) generates faster than any of the larger MoEs on M2 Ultra - 134 t/s versus 60-130 t/s - because less data has to stream per token. Second, the DGX Spark’s GB10 chip is faster at prompt processing but not always at single-batch generation: 4,506 prompt t/s on gpt-oss-20B versus 2,713 on M2 Ultra, but only 83 t/s versus 130 t/s for the same model’s generation.
| Hardware | Model | PP t/s @ PP=512 | Gen t/s @ PP=512, B=32 |
|---|---|---|---|
| M2 Ultra | gpt-oss-20B | 2,875 | 536 |
| M2 Ultra | Qwen2.5-Coder-7B | 1,637 | 611 |
| M2 Ultra | Qwen3-Coder-30B-A3B | 2,778 | 393 |
| M2 Ultra | gemma-3-4b-it-qat | 3,125 | 788 |
| M2 Ultra | GLM-4.7-Flash | 2,132 | 261 |
| M2 Ultra | gpt-oss-120B | 1,708 | 279 |
| DGX Spark | gpt-oss-20B | 4,767 | 681 |
| DGX Spark | gemma-3-4b-it-qat | 6,507 | 820 |
| DGX Spark | Qwen3-Coder-30B-A3B | 3,101 | 347 |
Source: same llama.cpp bench files. Continuous batching is what makes serving a local model to your household viable: generation t/s rises 3-8x with batch 32 on these models.
Apple Silicon at higher tiers
The community-run ml-explore/mlx discussion #3209 measured MLX performance on a Mac Studio M3 Ultra (512 GB) and Qwen 32B at every quant:
| Quant | Gen t/s @ 1K context | Gen t/s @ 32K context |
|---|---|---|
| F16 | 10.4 | 8.5 |
| Q8 | 18.3 | 13.4 |
| Q6 | 23.0 | 15.7 |
| Q4 | 31.2 | 19.0 |
| Q3 | 38.1 | 21.1 |
| Q2 | 47.6 | 24.1 |
Qwen 32B Q4 on M3 Ultra hits 31 t/s at 1K context, matching Qwen3-Coder-30B-A3B Q8_0 on M2 Ultra. M3 Ultra’s 512 GB RAM (vs 192 GB on M2 Ultra) lets you run 405B-class models at usable speeds, but per-token throughput still tracks bandwidth.
How Quantization Changes Speed
Lower-bit quants shrink the data movement per token, so they speed generation up. The ml-explore/mlx table is the cleanest example: Qwen 32B at 1K context goes from 10.4 t/s at F16 to 47.6 t/s at Q2, a 4.6x speedup from quantization alone. The trade-off is quality, which what quantization costs you covers.
Three caveats. First, gains shrink at longer context: the same Qwen 32B at 32K context goes from 8.5 t/s at F16 to 24.1 t/s at Q2 (2.8x), because attention compute grows with context length. Second, prompt-processing does not speed up as much with quant because prefill is compute-bound, not memory-bound. Third, AWQ/GPTQ often run slower than GGUF at the same nominal bit rate because the dequant kernel is heavier (which quantization format).
How Context Length Changes Speed
Generation t/s drops as the context window fills. The KV cache grows linearly with prompt length, and attention time grows quadratically. The llama.cpp M2 Ultra table shows the curve on real models:
| Model | t/s @ 2K | t/s @ 8K | t/s @ 32K |
|---|---|---|---|
| gpt-oss-20B MXFP4 | 130 | 117 | 98 |
| Qwen2.5-Coder-7B Q8_0 | 80 | 71 | 57 |
| Qwen3-Coder-30B-A3B Q8_0 | 79 | 65 | 45 |
| GLM-4.7-Flash Q8_0 | 60 | 40 | 22 |
| gpt-oss-120B MXFP4 | 86 | 78 | 66 |
Source: llama.cpp Mac M2 Ultra bench.
The MoE gpt-oss models hold up best: at 32K context, gpt-oss-120B still generates 66 t/s, while the dense GLM-4.7-Flash drops to 22 t/s. The ml-explore/mlx Qwen 32B numbers show the same effect: 31 t/s at 1K dropping to 19 t/s at 32K for Q4.
How to Make Local LLMs Faster
Six levers actually move tokens per second, in roughly descending order of impact.
- Pick a higher-bandwidth accelerator. A 5090 is 78% faster than a 4090 for memory-bound generation. M5 Ultra at 1.2 TB/s is roughly 2x faster than M5 Max at 460-614 GB/s.
- Quantize harder. Q4 is the usual sweet spot; Q3 and Q2 buy 20-50% more tokens per second at meaningful quality cost.
- Switch runners when one is faster. Ollama on Apple Silicon uses MLX by default; llama.cpp on NVIDIA uses CUDA; vLLM and TGI serve with continuous batching. The same model can run 1.5-3x faster on the runner with the best-tuned kernel for that backend (best local LLM runners).
- Batch concurrent requests. A single Ollama user gets one user’s throughput; an Open WebUI server with four active users gets closer to the batched numbers above.
- Use speculative decoding. A small draft model proposes tokens; the big model verifies them in one pass. llama.cpp supports draft-model, EAGLE-3, n-gram, DFlash, and DSpark variants; n-gram mod shows a 0.70 acceptance rate (llama.cpp speculative docs). Typical win is 1.5-3x on long generations where draft patterns repeat.
- Pick a model with grouped-query or multi-query attention. GQA reduces KV-cache reads per token, directly cutting generation latency. Llama-3, Qwen3, and Mistral families use GQA; older Llama-2 and Bloom do not.
The Bottom Line
A reasonable target for a single user on a current consumer setup is 30-130 tokens per second for generation: M2 Ultra or RTX 3090-class hardware at Q4 gets most 7-30B models into that range. A current high-end desktop (RTX 5090, M5 Ultra) gets you 60-130 t/s on the same models and can serve 200-500 t/s in aggregate under continuous batching. A 120B MoE at Q4 runs at 60-130 t/s on M2 Ultra but only at 8-22 t/s on most consumer cards, because it needs 60+ GB of memory bandwidth those cards do not have.
Speed mostly tracks memory bandwidth, not raw FLOPS, and quant at Q4 buys you 2-3x over F16 for the same hardware.
All eleven local-AI guides, by hardware tier and by task, are indexed at Local AI by VRAM: Which Models Fit Your GPU.