When a local model fails to load or a chat dies mid-session, memory is the first thing to check, because both major runtimes treat it as a distinct failure mode rather than a generic crash. Ollama keeps a list of error strings it explicitly matches as out-of-memory conditions, and llama.cpp’s allocators report a failed allocation with the size they asked for. Both also report the memory headroom they had when the load was attempted, so the diagnosis does not require guesswork.
Read the error before you change anything
| What you see | What it means | First move |
|---|---|---|
cudaMalloc failed in the CUDA log | The GPU refused a block. ggml’s CUDA allocator logs “allocating %.2f MiB on device %d: cudaMalloc failed: %s” (ggml-cuda.cu) | Cut context length, or push layers into system RAM |
failed to allocate Metal buffer of %zu bytes (out of memory) | The Metal allocator could not get a buffer of the size shown (ggml-metal.cpp) | Same as above, plus unload other models |
failed to allocate %s buffer of size %zu | A buffer of a named type could not be allocated, and the message gives the type and the size (ggml-backend.cpp) | Reduce the context or the model size, depending on which buffer failed |
failed to reserve graph buffers or failed to allocate graph | The compute buffers for the graph could not be reserved or allocated (ggml-backend.cpp). Ollama’s own matcher counts any “failed to allocate” line, and graph buffer reservation failures, as memory exhaustion (status.go) | Lower batch and ubatch size |
model is too large for system memory | Ollama’s scheduler gave up because, with no other model loaded and partial offload to system RAM allowed, the model still will not fit (sched.go) | Use a smaller model or a smaller quantization |
llama-server model predicted to exceed available memory, evicting | Other models were loaded and the new one was predicted to need more than 80% of the free memory (sched.go) | Unload the other model first |
Two Ollama log lines settle most arguments before you touch a setting. On every load it reports “system memory” with total, free and free swap, and “gpu memory” with the available, free, minimum and overhead figures it used for the decision (sched.go). The same file also logs “llama-server load failed; reducing automatic context and retrying once”, which is Ollama shrinking an automatic context window and trying again on its own before giving up.
Ollama: the settings that move the number
| Setting | Default | What it changes | The catch |
|---|---|---|---|
OLLAMA_CONTEXT_LENGTH | Set by VRAM tier at startup | Context window, and with it the KV cache | The FAQ still documents 4096 as the default, while the code sets 262144 at or above 47 GiB of total VRAM (less any OLLAMA_GPU_OVERHEAD), 32768 at or above 23 GiB, and 4096 otherwise (routes.go). Set it explicitly rather than assuming |
OLLAMA_NUM_PARALLEL | 1 | Parallel requests per model | The FAQ states “Required RAM will scale by OLLAMA_NUM_PARALLEL * OLLAMA_CONTEXT_LENGTH” (FAQ) |
OLLAMA_MAX_LOADED_MODELS | 3 times the number of GPUs, or 3 on CPU | How many models stay resident at once | Every resident model holds its own VRAM or RAM |
OLLAMA_KEEP_ALIVE | 5 minutes | How long a model stays loaded | The catch is reload time, not correctness. ollama stop <model> frees it now |
OLLAMA_KV_CACHE_TYPE | f16 | Precision of the KV cache | q8_0 uses roughly half the memory of f16, q4_0 roughly a quarter, per the FAQ. It requires Flash Attention, and the FAQ warns that models with a high GQA count lose more precision than models with a low count |
OLLAMA_FLASH_ATTENTION | On automatically where the backend supports it | Memory used as context grows | It is already on where it can be, and the KV cache quantization in the row above needs it enabled |
OLLAMA_GPU_OVERHEAD | 0 bytes | VRAM reserved per GPU | Set it when another application is holding VRAM that Ollama should not try to use |
The FAQ is explicit about one consequence that catches people out: “Parallel request processing for a given model results in increasing the context size by the number of parallel requests. For example, a 2K context with 4 parallel requests will result in an 8K context and additional memory allocation” (FAQ). On GPU it is stricter still, since “new models must be able to completely fit in VRAM to allow concurrent model loads”.
llama.cpp: the flags that move the number
| Flag | Default | Effect | The catch |
|---|---|---|---|
-c, --ctx-size N | 0, meaning the model’s own context | Context window | 0 means loaded from model, so the fitter may change it |
-ngl, --gpu-layers N | auto | Layers held in VRAM | A number, auto or all. Lowering it moves layers to system RAM and slows generation |
-ctk, --cache-type-k and -ctv, --cache-type-v | f16 | KV cache precision. Allowed values are f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1 | Quantizing the cache saves memory at some cost in precision |
-kvo, --kv-offload / -nkvo, --no-kv-offload | Enabled | Whether the KV cache lives on the GPU | Turning it off moves the cache to system RAM |
-np, --parallel N | -1, meaning auto | Server slots | With auto slots the KV cache is one unified buffer shared across all of them (-kvu), and --kv-unified-per-slot N sizes that pool to slots times N, so more slots can mean more memory |
-b, --batch-size and -ub, --ubatch-size | 2048 and 512 | Prompt processing buffers | The compute buffer is reserved from these, and it is what the graph allocation failure refers to |
-fit, --fit | on | ”whether to adjust unset arguments to fit in device memory” (arg.cpp) | It only changes parameters still at their defaults, and the context size only when it is 0 |
-lm, --load-mode mlock | auto | Keeps the model in RAM “rather than swapping or compressing” (server README) | Locks the RAM, so nothing else can have it |
That last row matters on machines with no discrete GPU. llama.cpp memory-maps the weights by default, which the server README describes as “mmap, unless a device does not support it”, so under memory pressure the kernel can drop weight pages and read them back from disk later. Forcing mlock keeps the model resident, at the cost of tying up RAM.
llama.cpp also ships a standalone fitter, llama-fit-params. Running it with -fitp on prints the estimate it will use, broken down as “device, model, context, compute” in MiB, and without that flag it prints the fitted arguments it would use, starting with -c and -ngl (fit-params.cpp). It answers the question the error message only hints at: which term is the one that does not fit.
The order that works
- Read the log, not the message. The runtime tells you which term failed.
- Cut the context window first. The KV cache is sized from it, the Ollama FAQ notes that required RAM scales with it, and cutting it costs you history length.
- Quantize the KV cache if the context still does not fit.
- Set parallel requests to one.
- Unload everything else, including the models Ollama keeps warm for five minutes.
- Move layers into system RAM (
-nglbelow the layer count) if the model must stay the same size. - Drop to a smaller quantization or a smaller model.
- Add RAM, last, because it is the only step that leaves the software side untouched.
When the model is genuinely too big
At some point the arithmetic does not close, and no setting fixes it. The weights plus a usable context plus compute buffers have to fit in the card, and the VRAM tier guides are built on that. If the model you want is one tier up, the honest options are a lower-bit quantization, which the format comparison covers, or a smaller model from the same family. The breakdown of where the memory actually goes, including why the published file size is only one of four terms, is set out in how much VRAM a local LLM needs.
Bottom line
Memory failures are diagnosable, because both runtimes report the term that failed and the headroom they had. Cut context before anything else, quantize the KV cache next, cut parallel requests to one, unload the models you are not using, and only then decide the model is too big. The order is by cost: cutting context costs history, quantizing the KV cache costs a little precision, moving layers to system RAM costs speed, and only a smaller quantization or a smaller model changes the model you are running.
For more on the local AI beat, see our local AI coverage.