The number on a model’s library tag is the weights file. The number on the VRAM tier guides is also the weights file. Both are correct for what they list, but VRAM a model actually uses at run time is weights plus three more terms: the key-value (KV) cache for your context length, framework overhead, and (when serving multiple people) per-user cache copies. Treating the published size as the load-bearing figure is how a chat session loads cleanly on the first prompt and dies on the second.
What Actually Adds Up
Four terms, each with a different source.
| Term | Order of magnitude | Source |
|---|---|---|
| Model weights | tens of GB | the GGUF or safetensors file on the model card |
| KV cache for your context | 0.1 GB to tens of GB | model architecture and context length |
| Framework overhead | 0.3 GB to 5 GB | the runtime: Ollama, vLLM, llama.cpp, Transformers |
| Per-user cache copies | same as one user’s cache | number of concurrent requests |
The Hugging Face tutorial on LLM inference optimization measured bigcode/octocoder at 29.0 GB peak against a back-of-envelope prediction of 31 GB at bf16, 15.2 GB at 8-bit (close to the 15.5 GB prediction), and 9.5 GB at 4-bit (above the 7.75 GB prediction), with single-token inference over a 60-token prompt. The actual measurements sat within roughly 2 GB of the back-of-envelope in every case - the formula was within a few percent at bf16 and 8-bit, and within ~1.7 GB at 4-bit. Framework overhead from the runtime (Ollama, vLLM, llama.cpp, Transformers) is small but real and should be budgeted at roughly 0.5 GB above whatever the formula returns; treat it as a flat addition rather than a percentage of weights.
The KV Cache Formula
Per token, in bytes:
2 x num_hidden_layers x num_key_value_heads x head_dim x bytes_per_elementThe tutorial spells the same formula in prose: the cache holds key and value vectors for every layer and every KV head across the whole generated sequence, so two tensors multiplied by the cache positions. The model exposes num_key_value_heads rather than the query-head count, because every grouped-query model has fewer KV heads than query heads, and only the KV heads get cached. The Hugging Face paper page for the GQA paper describes the trade-off; multi-query attention goes further with one KV head total, which is what older MQA models (Falcon, PaLM, BLOOM) carry.
The bytes-per-element term is what changes with the runtime flag:
- fp16 or bf16: 2 bytes per element
- Q8 KV cache (vLLM, llama.cpp on supported builds): 1 byte per element
- Q4 KV cache (llama.cpp on some builds): 0.5 bytes per element
Three real configurations make the math concrete.
Qwen3-32B
From the Qwen3-32B config, read 2026-08-25: 64 layers, 64 query heads and 8 KV heads, head_dim 128, bfloat16 dtype. The model card lists 32.8B parameters total and 31.2B non-embedding. License: Apache-2.0.
KV cache per token at fp16 = 2 x 64 x 8 x 128 x 2 = 262,144 bytes = 256 KB.
| Context length | KV cache | Ollama Q4_K_M weights | Approximate total VRAM |
|---|---|---|---|
| 4,096 tokens | 1.0 GB | 20 GB (Ollama tag, read 2026-08-25) | 21.5 GB |
| 8,192 tokens | 2.0 GB | 20 GB | 22.5 GB |
| 16,384 tokens | 4.0 GB | 20 GB | 24.5 GB |
| 32,768 tokens (native) | 8.0 GB | 20 GB | 28.5 GB |
| 131,072 tokens (YaRN extended) | 32.0 GB | 20 GB | 52.5 GB |
Qwen3-32B at Q4_K_M fits a 24 GB card up to 8k context, runs tight at 16k (the table lands at 24.5 GB), and becomes a multi-GPU or offloaded problem from 32k upward.
Qwen3-30B-A3B (mixture-of-experts)
A counter-example where the published GGUF size is the worst case, not the runtime cost. From the Qwen3-30B-A3B config, read 2026-08-25: 48 layers, 32 query heads and 4 KV heads, head_dim 128, 128 experts with 8 active per token. License: Apache-2.0, license tag confirmed via the Hugging Face model API.
KV cache per token at fp16 = 2 x 48 x 4 x 128 x 2 = 98,304 bytes = 96 KB.
| Context length | KV cache | Ollama Q4_K_M weights | Approximate total VRAM |
|---|---|---|---|
| 4,096 tokens | 0.4 GB | 19 GB (Ollama tag, read 2026-08-25) | 19.7 GB |
| 8,192 tokens | 0.75 GB | 19 GB | 20.1 GB |
| 16,384 tokens | 1.5 GB | 19 GB | 20.8 GB |
| 32,768 tokens | 3.0 GB | 19 GB | 22.3 GB |
The 30B-A3B file is around 19 GB on the Ollama tag page, but only 3.3B of the 30.5B total parameters are active for any one token, so what controls throughput is the active-parameter count, not the file size on disk. The model card states 30.5B total and 3.3B activated. This is the gap between “what the file size says” and “what VRAM you need” in its cleanest form.
What Changes the Answer
Three knobs move the cache without changing the model.
- Context length is the steepest. Cache scales linearly with it, so a 32k chat session is roughly 8x the cache of a 4k session in the same model. Long-document summarisation and code-repo work are the workloads that eat VRAM, even on small models.
- Batch size matters when serving multiple people. Each concurrent request holds its own KV cache, so ten users on a 4k context each equals forty users on a 1k context in cache terms. The local serving guide handles this separately; the safe household pattern is to bind to the LAN and rate-limit.
- Cache quantisation changes the bytes-per-element term. vLLM and llama.cpp both support Q8 and Q4 KV caches on a growing number of builds; on a 32k-context Qwen3-32B this halves or quarters the 8 GB cache line, so 4 GB or 2 GB instead.
The published file size does not move with any of those knobs. The published “X GB” on a tier guide is the lower bound, not the answer. The cluster’s hub page links eighteen guides by tier; this page is the arithmetic those tiers sit on.
What This Means
A rough rule of thumb until the model has unusually large KV head dimensions: weights plus 25% of the weight file for a 32k context, plus 10% for 8k, plus 5% for 4k. Re-compute with the formula when the figures stop adding up; it’s a small handful of variables and the model card lists them. The quantisation guide discusses what Q4_K_M costs in quality, separately from what it saves in size, so the size half is here and the quality half is on that page.
A further catch: Mixture-of-Experts models like the 30B-A3B above advertise total parameter counts that look like a different size class than the active parameters suggest. Read both numbers from the model card before you budget. A 19 GB Ollama tag is not a “fits on 16 GB” model, because a 19 GB file still needs more than its own size in VRAM to load with full context; it is a “use the 24 GB tier” model that is faster than its file size implies.
The Bottom Line
The four-term picture is the one to keep: weights + KV cache + framework overhead + per-user copies. Compute the cache from the model config, not from a tier page, and add at least one gigabyte of framework allowance to whatever comes out. The result is the VRAM your session will actually use, before you try to fit a 32k chat on a card that listed a model and stopped reading.