A model card says “262K context window”. You load the model on a 16 GB GPU. You ask a question with a 200K-token PDF attached. The session dies or the runner quietly falls back to CPU and crawls. The advertised number was real. It was also not the number you needed.
Two numbers hide behind “context window”. One is what the model was trained to handle at full attention quality. The other is how many tokens the runtime can fit alongside the weights on your hardware. They diverge fast: the same 32B model that ships with a 40,960-token training context can advertise 128K or higher on the card after a RoPE-extension fine-tune, while the practical ceiling on a 24 GB card is closer to the trained number once you account for the KV cache.
This page is about the practical number, the math it falls out of, and the levers that change it.
What “context” is, mechanically
When a transformer reads a prompt, every layer produces two tensors per token: a key and a value, together the “KV pair”. During generation the model has to look at every previous token’s KV pair to score the next one. Recomputing those pairs at every step is wasteful, so every runtime caches them. The cache grows linearly with the number of tokens you have in play, and every layer keeps its own copy. The Hugging Face optimization guide walks through this: at any input length, the cache size is roughly 2 × number of layers × number of KV heads × head dimension × bytes per element, once per token (HF optimization guide).
Two modern tricks shrink that cache without retraining the model:
- Grouped-Query Attention (GQA). Most layers have far fewer key-value heads than query heads, because the value tensor is a low-rank projection that several queries can share. Llama 2 70B and Llama 3 use 8 KV heads against 64 attention heads; Qwen3-32B uses 8 KV heads against 64. Fewer heads means a smaller cache, and the paper shows the quality cost is small (Ainslie et al., 2023).
- Multi-Query Attention (MQA). The aggressive version: one KV head total. Falcon and PaLM use it. Cache shrinks by another factor of
n_head, but quality takes a small hit.
The cache dtype matters as much as the head count. FP16 is the default for both keys and values in llama.cpp’s server; switching to Q8 halves the cache, Q4 quarters it. The cost is a small accuracy loss, and for long-context retrieval most readers never notice it.
The arithmetic, on three real models
The Hugging Face configs below are the load-bearing inputs. Every figure below is computed from them.
| Model | Layers | KV heads | head_dim | max_position_embeddings | bytes / token (FP16) | Cache at max advertised context |
|---|---|---|---|---|---|---|
| Qwen3-32B | 64 | 8 | 128 | 40,960 | 256 KB | ~10.0 GB |
| Qwen3.5-9B | 32 | 4 | 256 | 262,144 | 128 KB | ~32.8 GB |
| Qwen3-235B-A22B-Instruct-2507 | 94 | 4 | 128 | 262,144 | ~188 KB | ~48.2 GB |
Sources: Qwen3-32B config, Qwen3.5-9B config, Qwen3-235B-A22B-Instruct-2507 config. The “bytes / token” column is 2 × layers × kv_heads × head_dim × 2 for FP16.
Two things jump out of that table. First, the cache grows linearly with context, and at a published ceiling it can easily exceed the weight file. Qwen3.5-9B at 262K context needs ~33 GB for the cache against ~6.6 GB for a Q4_K_M weight file in Ollama. Second, the trained maximum is usually the honest maximum: 40,960 for Qwen3-32B, which is what the model saw during pretraining. Anything beyond that is extrapolation, not free.
What “extrapolation” actually buys you
Most modern LLMs use Rotary Position Embeddings (RoPE) rather than absolute position vectors. RoPE encodes position by rotating the query and key vectors, and the rotation has a base frequency rope_theta baked into the config (Su et al., 2021). When a token sits beyond the trained context, the rotation angle is larger than anything the model has seen, and the geometry starts to fall apart. That’s why naive extrapolation produces nonsense.
Two engineering responses exist:
- Fine-tune the model at longer context with adjusted
rope_theta. The model card’s 128K, 200K or 1M number is the result of a continued-pretraining run, not the original training. Qwen3.5-9B’s card reports a 262K native context with YaRN-style extension to 1,010,000. Quality at the extension end is typically lower than quality at the trained end; the model card and benchmarks are at the trained length. - YaRN-style frequency interpolation. YaRN rescales the rotation frequencies so that the long-context rotation angle maps back onto the trained range, then fine-tunes for a small fraction of the original compute (Peng et al., 2023). The headline number from the paper: “10x less tokens and 2.5x less training steps than previous methods.” A LLaMA-2 7B model trained at 4K can reach 128K with this approach. The trade-off is that long-context retrieval and long-context faithfulness degrade at the very top of the extended range; benchmarks usually quote results at the trained extension length, not the absolute maximum.
Both responses leave you with the same practical problem: a model that is advertised at 262K and trained at 32K-128K is more reliable at 32K than at 260K. Treat the card as the upper bound and the trained number as the comfortable one.
What your hardware can actually serve
The cache grows with tokens, the weights are fixed, and both have to live in VRAM (or in unified memory on Apple Silicon, which has the same constraint under a different name). Add the framework’s overhead (a few hundred MB to a few GB depending on the runner, see how much VRAM does a local LLM actually need) and you get a ceiling.
Worked example, Qwen3-32B Q4_K_M on a 24 GB card:
- Weights: ~20 GB (the Ollama library tag reports 20 GB).
- KV cache at 32K context, FP16: ~8.0 GB.
- Total at 32K: ~28 GB. Already past 24 GB.
- KV cache at 16K context, FP16: ~4.0 GB.
- Total at 16K: ~24 GB. Tight; over budget by the framework overhead.
Drop the KV cache to Q8 and the same model fits 32K context with cache quantized, on a 24 GB card with thin margin. That is the practical configuration for that hardware, not the 40,960-token “context window” the card advertises. llama.cpp’s server README documents --cache-type-k q8_0 and --cache-type-v q8_0 for exactly this case (llama.cpp server README).
For a small model on modest hardware, the math is friendlier. Qwen3.5-9B Q4_K_M is ~6.6 GB of weights, and at 32K context the FP16 KV cache is ~4.0 GB. On a 12 GB card, 32K is feasible with cache quantization; on 16 GB it is comfortable at FP16. Beyond 64K context, even 16 GB gets tight without Q8 or Q4 KV cache.
Two levers, named explicitly:
- Context length (
--ctx-sizein llama.cpp’s server). Smaller is faster and cheaper. The model card is the ceiling, not the floor. - KV cache quantization (
--cache-type-k,--cache-type-v). Halves or quarters the cache cost. Default is FP16; Q8 is usually a free lunch; Q4 occasionally shows up in long-context retrieval drops.
The HF transformers library also exposes cache_implementation="quantized" and "offloaded", which move the cache to CPU or quantize it via HQQ or Quanto backends (HF KV cache guide). Offloading keeps the GPU free for weights and is the right choice on a small card with system RAM to spare; throughput drops because every cache read crosses the PCIe boundary.
The Bottom Line
The card number is the trained-after-extension ceiling, not what your card will run. The honest ceiling is roughly advertised context × 256 KB per token × number of KV layers for a Qwen3-class GQA model in FP16, less whatever you save by quantizing the cache. Treat published context windows the same way you treat published VRAM requirements: as the upper bound under a specific set of assumptions, not a guarantee on your hardware.
Practical defaults that hold across models:
- 4K to 8K is the safe length for everyday chat on any 16 GB or larger setup. Quality is highest here because no extension or RoPE trick is in play.
- 16K to 32K is where most local RAG setups live. Cache is real but manageable; cache quantization is usually a free win.
- 64K and above is where the cache stops being a rounding error. Plan for it; expect the trained length to feel better than the extended length; benchmark your own workload before you trust the card.
If you have a specific context length in mind, get the model’s config.json from Hugging Face, compute 2 × num_hidden_layers × num_key_value_heads × head_dim × 2 bytes per token, multiply by your target context, and add the weight file size. If the sum exceeds your VRAM, either shorten the context, quantize the cache, or pick a smaller model.