Every guide on this site so far starts from one number: how much VRAM your graphics card has. That leaves out office desktops, thin laptops, mini PCs and most rented virtual machines, none of which have a discrete GPU at all, and all of which can still run a useful open-weight model.
Verified August 2026 against the runtimes’ own documentation and the model repositories themselves.
Nothing About Local Inference Requires a GPU
llama.cpp, the engine underneath most local AI software, describes itself as a “plain C/C++ implementation without any dependencies” and lists “AVX, AVX2, AVX512 and AMX support for x86 architectures” alongside its CUDA, Vulkan and Metal backends. It is MIT-licensed. GPU acceleration is one option among many in that project, not a prerequisite.
The same holds one layer up. Ollama, also MIT-licensed, documents an environment variable for the opposite of what most people want: its GPU documentation says that “if you want to ignore the GPUs and force CPU usage, use an invalid GPU ID (e.g., -1)”. The same page sets the NVIDIA floor at “compute capability 5.0+ and driver version 550 and newer”, which is the line a lot of older and integrated hardware falls below. Below that line the work goes to the processor, and it still works.
What changes without a GPU is not whether a model runs. It is which models are worth loading, and how patient you have to be.
Size the Model to Free RAM, Not Total RAM
On a CPU-only machine the model weights sit in system memory, competing with the operating system and everything else you have open. Two figures matter and they are not the same: the download size, which is published, and the runtime footprint, which is larger because the context window is allocated on top of the weights. Ollama’s FAQ states that “required RAM will scale by OLLAMA_NUM_PARALLEL * OLLAMA_CONTEXT_LENGTH”, so a long context and concurrent requests both add memory on top of the number in the table below.
Sizes are the Ollama library’s published tag sizes; licences are read per repository from Hugging Face, never inferred from the model family.
| Model | Ollama tag | Download size | Licence | Note |
|---|---|---|---|---|
| Qwen3.5 0.8B | qwen3.5:0.8b | 1.0GB | Apache-2.0 | Fits anywhere, including a 4GB machine |
| Gemma 3 1B | gemma3:1b | 815MB | gemma | Custom licence, read it before commercial use |
| Llama 3.2 3B | llama3.2:3b | 2.0GB | llama3.2 | Community licence, HF repo is gated |
| Qwen3.5 2B | qwen3.5:2b | 2.7GB | Apache-2.0 | The sensible default on 8GB of RAM |
| Gemma 3 4B | gemma3:4b | 3.3GB | gemma | Gated on Hugging Face, pulls freely from Ollama |
| Qwen3.5 4B | qwen3.5:4b | 3.4GB | Apache-2.0 | Comfortable on 16GB |
| Qwen3.5 9B | qwen3.5:9b | 6.6GB | Apache-2.0 | The upper end of dense-on-CPU |
The Gemma row carries a trap this site has published wrongly before. google/gemma-3-4b-it is marked gated: manual on Hugging Face, meaning a manual access request, while the Ollama tag downloads without one. The Llama 3.2 repository is gated the same way. If you plan to move from a runtime pull to the original weights later, that access request is waiting for you.
Mixture-of-Experts Is the CPU Trick Worth Knowing
Generation speed on a CPU tracks how much of the model has to be read for each token, and mixture-of-experts models are built so that most of the model is not read. Qwen3.5-35B-A3B is “35B in total and 3B activated”, with 256 experts of which “8 Routed + 1 Shared” are active per token. Granite 4.0 H Tiny is 7B total and 1B active, from 64 experts with 6 active per token, under Apache 2.0.
| Model | Ollama tag | Download size | Total / active parameters | Licence |
|---|---|---|---|---|
| Granite 4.0 H Tiny | granite4:7b-a1b-h | 4.2GB | 7B / 1B | Apache-2.0 |
| Granite 4.0 H Small | granite4:32b-a9b-h | 19GB | 32B / 9B | Apache-2.0 |
| Qwen3.5 35B-A3B | qwen3.5:35b-a3b | 24GB | 35B / 3B | Apache-2.0 |
The catch is the one people miss: the whole model still has to be in memory. qwen3.5:35b-a3b is a 24GB download whether or not only 3B of parameters are touched per token, so it needs a 32GB or 64GB machine even though its per-token work resembles a 3B model. That trade, lots of cheap RAM instead of scarce VRAM, is the one case where a CPU-only box with plenty of memory is not obviously worse off than a mid-range graphics card. On the systems where it fits, granite4:7b-a1b-h at 4.2GB is the version of the same idea that runs on ordinary hardware.
Prompt Processing and Generation Fail Differently
These are two separate jobs with two separate bottlenecks, and llama.cpp’s own build documentation makes the split explicit. Building with a BLAS library “may lead to some performance improvements in prompt processing using batch sizes higher than 32”, and then, plainly: “Using BLAS doesn’t affect the generation performance.”
The practical reading is that reply speed and the wait before the first word come from different places. Pasting a long document, a large system prompt or a full chat history in as context is the expensive part on a CPU, and it is charged again whenever the cached prefix is invalidated. Short back-and-forth chat is the shape of workload that suffers least. Retrieval over a document set, where every query drags in fresh chunks of text, is the shape that suffers most.
This site does not publish tokens-per-second figures for CPU inference, because it does not own the hardware to measure them on and second-hand numbers for a machine you do not have are worth nothing. Measure your own: llama.cpp ships a llama-bench tool in its tools directory for exactly this, and it reports prompt-processing and generation rates separately.
Your Integrated GPU May Count
“No GPU” usually means no discrete GPU. Integrated graphics may still be usable, because llama.cpp lists Vulkan and SYCL among its supported backends, and Ollama’s GPU documentation says Vulkan “is enabled by default when the backend is installed” on Windows and Linux, with Intel and AMD driver instructions given for Linux.
Two warnings from that same page. Ollama notes that Vulkan needs elevated capability or root to report available VRAM, and without it “Ollama will use approximate sizes of the models to make best effort scheduling decisions”. And it acknowledges instability directly, telling users on mixed systems where “the Vulkan iGPU is unstable” to point GGML_VK_VISIBLE_DEVICES at the discrete card instead. An integrated GPU also carves its memory out of the same system RAM the model is using, so it moves work rather than adding capacity.
Apple silicon is a different case again and not really a CPU-only one: llama.cpp treats it as “a first-class citizen”, optimised via ARM NEON, Accelerate and Metal. If that is your machine, running LLMs on Apple silicon is the relevant guide.
What to Decide Before You Download
Pick the quantization deliberately. llama.cpp supports “1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization”, and the default tags in the table above are mostly 4-bit builds. On a CPU the temptation is to drop lower to fit a bigger model into the same RAM, which is the exact trade covered in which quantization format to use.
Expect memory pressure to show up as queueing rather than an error. Ollama’s FAQ states that when there is not enough memory for a new model while others are loaded, “all new requests will be queued until the new model can be loaded”. A local assistant that has apparently hung is often a memory problem, not a crash.
And if you do end up buying hardware, the VRAM tier index is the other half of this decision: it starts from a card and works down to the models, which is the reverse of this page.
Bottom Line
A machine with no discrete GPU runs local AI today, and the software is not a special CPU-only fork - it is the same MIT-licensed llama.cpp and Ollama everyone else runs. On 8GB of RAM, qwen3.5:2b at 2.7GB is the realistic default. On 16GB, qwen3.5:4b at 3.4GB or granite4:7b-a1b-h at 4.2GB. On 32GB or more, the mixture-of-experts models are the best use of memory you will not otherwise be spending on VRAM.
The honest limits are the ones this page has named rather than the ones a benchmark chart would show: long contexts are the expensive operation, retrieval-heavy work is the worst fit, integrated-GPU offload is documented by its own maintainers as sometimes unstable, and the only throughput number worth trusting is the one you measure on the machine in front of you.