Every request that re-sends a long prompt makes the runner recompute work it has already done, unless the runner recognises the identical prefix and reuses what it computed last time. Prompt caching is the mechanism for that, and every major local runner implements some version of it. What differs is how it is switched on, what it reports back, and how much it costs you in memory.
What prompt caching actually saves
The unit that gets reused is the KV cache: the key and value vectors computed for every token read, for every layer, for every key-value head. Its size is fixed by the architecture and the context length, which is why the VRAM arithmetic page is about memory, not time.
Reuse does not shrink that cache. It avoids recomputing it. If a 20,000-token system prompt arrives again with 300 new tokens appended, an uncached runner processes 20,300 tokens of prompt; a caching runner processes 300. Prompt ingestion and token generation run at different speeds, so the saving is usually larger than a naive token count suggests, but it lands entirely in latency, not in VRAM.
Two settings look alike and solve opposite problems. Reuse buys you time. Cache quantization buys you memory. The llama.cpp server takes f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0 and q5_1 for both the key and the value cache through -ctk/--cache-type-k and -ctv/--cache-type-v, defaulting to f16 on both. Ollama exposes the same trade in its FAQ, where OLLAMA_KV_CACHE_TYPE defaults to f16, with q8_0 described as using “approximately 1/2 the memory of f16 with a very small loss in precision” and q4_0 as “approximately 1/4 the memory of f16 with a small-medium loss in precision”.
What each runner does
| Runner | Cache-reuse control | Default | Reports reuse per request | Notes |
|---|---|---|---|---|
llama.cpp llama-server | --cache-prompt / --no-cache-prompt, plus --cache-reuse N | --cache-prompt enabled, --cache-reuse 0 | Yes: tokens_cached in the response | --cache-reuse needs a chunk size; without it the KV shift pass is off |
| Ollama | Not exposed as a per-request control in its documentation | Not stated | Not stated in the FAQ or context-length pages | Docs cover context length and KV cache dtype, not the reuse mechanism |
| vLLM | enable_prefix_caching (CLI --enable-prefix-caching / --disable-prefix-caching) | True in current vllm/config/cache.py | Not stated on the docs page | Docs say to set it; the current source default is already on |
| LM Studio (llama.cpp engine) | Not exposed; controlled through load config | Not stated | Not stated | offload_kv_cache_to_gpu and flash_attention are engine-scoped |
| LM Studio (MLX engine) | Automatic KV cache checkpointing | Automatic | Yes, by cached token count in the vendor’s own benchmark | Disk-backed, lives only while the model is loaded |
Two rows deserve reading closely.
On llama.cpp, the per-request cache_prompt field is documented as re-using the KV cache from a previous request “if possible”, where “the common prefix does not have to be re-processed, only the suffix that differs between the requests”, and it defaults to true. The separate --cache-reuse N flag is what makes non-identical prefixes partially reusable: it is described as the “min chunk size to attempt reusing from the cache via KV shifting, requires prompt caching to be enabled”, and it defaults to 0, so the shift pass is off unless you raise it. The server also returns a tokens_cached value, defined as the “Number of tokens from the prompt which could be re-used from previous completion”, and streams a prompt_progress object containing total, cache, processed and time_ms when you ask for progress. That is the number to watch.
On vLLM, the documentation page states “Automatic Prefix Caching (APC in short) caches the KV cache of existing queries” and instructs you to “Set enable_prefix_caching=True in vLLM engine to enable APC”. The current source disagrees about the starting point: vllm/config/cache.py declares enable_prefix_caching: bool = True. In other words, on current vLLM the feature is already on unless you pass --disable-prefix-caching, and the docs page describes the older opt-in posture. Treat the source file as the current truth and the docs page as stale on this one point. The same page notes a --prefix-match-unit knob that controls “the granularity at which prefix-cache keys are computed” and suggests “64 is a reasonable starting point”, and it is explicit about the limit: prefix caching does not shorten decoding, so it buys almost nothing for a long generated answer.
Ollama is the honest gap in this table. Its documentation covers default context length by VRAM band and KV cache data type, but neither the FAQ nor the context-length page describes how prompt caching works or reports cached token counts. Rather than infer behaviour from silence, this page states what was searched: https://docs.ollama.com/faq and https://docs.ollama.com/context-length, both read on 2026-10-23.
LM Studio splits by engine. Its REST load API exposes context_length, described as the “Maximum number of tokens that the model will consider”, plus offload_kv_cache_to_gpu (“Whether KV cache is offloaded to GPU memory. If false, KV cache is stored in CPU memory/RAM”) and flash_attention. Both of the last two carry the same restriction in the documentation: “Will only have an effect on LLMs and decision models loaded by LM Studio’s llama.cpp-based engine.” On Apple Silicon the MLX engine does something different again. Writing in LM Studio’s blog, Neil Mehta describes a version that “saves KV cache records at fixed 256-token boundaries, then restores the longest available cached prefix”, so “By saving and restoring prompt cache to disk, the KV cache for follow-up requests does not need to be recomputed.” That store is temporary: it uses one scratch file under /tmp, and the metadata lasts only while the model stays loaded.
How to tell whether reuse is happening
Start with the number, not the intuition. On llama.cpp, read tokens_cached from the completion response, or request progress and watch the cache field in prompt_progress climb toward total. A run where cache stays near zero is re-processing the whole prompt every turn, which is the normal outcome when something early in the prompt changes between requests.
Audit what sits at the front of the prompt. Reuse only survives an identical prefix. A timestamp, a session ID, a retrieved-document block or a per-request system prompt all sit near the front and destroy the shared span. Stable material belongs first, per-request material last.
The memory side, measured
Cache reuse does not free memory, so the ceiling still comes from the KV cache arithmetic. Taking Qwen/Qwen3-8B as the worked example: the model API reports an apache-2.0 licence, gated: false, and 8,190,735,360 safetensors parameters, last modified 2025-07-26. Its config.json gives 36 hidden layers, 8 key-value heads and head_dim 128. At f16, that is 2 x 36 x 8 x 128 x 2 = 147,456 bytes, or 144 KiB, of cache per token.
| Context length | KV cache at f16 | Same at q8_0 (roughly half, per the Ollama FAQ) |
|---|---|---|
| 4,096 tokens | 576 MiB | about 288 MiB |
| 32,768 tokens | 4.5 GiB | about 2.25 GiB |
| 131,072 tokens | 18 GiB | about 9 GiB |
This is why the default matters more than the reuse setting. Ollama’s context-length documentation states that “Ollama defaults to the following context lengths based on VRAM”, giving 4k below 24 GiB, 32k between 24 and 48 GiB, and 256k at 48 GiB or above. Raising the ceiling is what costs you the memory; caching is what stops you paying to re-read it. The context-window comparison covers which models reach which lengths, and the runner comparison covers which engine to pick for the hardware.
Where the catch is
llama.cpp’s server documentation is candid about one. Enabling prompt caching “can cause nondeterministic results”, because depending on the backend the logits are “not guaranteed to be bit-for-bit identical for different batch sizes”, where one batch size is prompt processing and the other is token generation. Comparing model outputs across runs is reason enough to switch reuse off.
Three more. Ollama’s OLLAMA_KV_CACHE_TYPE is a global setting, so it applies to every model the server hosts rather than per model. llama.cpp’s unified KV buffer, on by default when the slot count is automatic, shares one pool across every sequence, so reuse and concurrent capacity compete for the same memory; --parallel and --cache-ram interact. And LM Studio’s KV cache settings silently do nothing outside its llama.cpp-based engine, which reads as a bug in the MLX path when it is a documented scope limit.
Bottom line
Prompt caching is not a feature you shop for. On current vLLM it is already on, on llama.cpp the per-request switch is already on and the more useful KV-shift pass needs --cache-reuse raised from its default of zero, and LM Studio’s MLX engine does it unasked. The practical work is verifying it: read tokens_cached, keep the shared prefix stable, and remember that reuse buys latency while KV cache quantization buys memory. Two problems, two knobs.