The biggest mistake people make with local AI is grabbing the largest open-weight model they can find, watching it crawl on their laptop, and giving up. The right model for you is the smallest one that answers your real questions at a speed you can live with. Choosing well takes about twenty minutes and saves weeks of frustration.
The four filters that actually matter are your hardware, the quantization label, the model’s intended task, and the model’s license. Everything else is noise you can ignore on your first try.
1. Map your hardware before you pick a model
Local inference needs memory more than anything else. Different tools give you different rules of thumb.
LM Studio’s system requirements recommend at least 16 GB of system RAM on Windows and Linux, and say an 8 GB Apple Silicon Mac can still work with smaller models and modest context windows. The same page notes that LM Studio’s x64 Windows build requires AVX2 instruction support, that Intel Macs are not supported, and that Linux is distributed as an AppImage and tested on Ubuntu 20.04 or newer.
Ollama’s GPU documentation explains that NVIDIA acceleration needs compute capability 5.0 or higher with driver 550 or newer, AMD uses ROCm v7, and Apple Silicon uses Metal. The FAQ adds an important caveat: “new models must be able to completely fit in VRAM to allow concurrent model loads.” When a model exceeds available VRAM, Ollama shows partial splits in ollama ps (for example “48%/52% CPU/GPU”) and the model runs but more slowly, with the disk or system RAM picking up the overflow.
A practical way to start: measure how much free memory your machine has when it is idle, then halve it. That is roughly the budget for the model plus its working context. Anything beyond that starts swapping to disk and the experience falls apart.
For a worked example on a specific tier, see our local model guide for 8 GB of VRAM.
2. Read the quantization label, not just the model name
Two downloads of the same 7B model can differ by 60 percent in file size because of how aggressively they were compressed. The label that ends the filename tells you which.
Hugging Face’s GGUF documentation lists the quantization types used by llama.cpp-compatible tools. Q4_K_M packs each weight at 4.5 bits, Q5_K_M at 5.5 bits, Q6_K at about 6.5 bits, and Q8_0 at 8 bits. The Llama project’s llama.cpp README confirms the same range, adding that the project supports 1.5-bit to 8-bit integer quantization for faster inference and reduced memory use.
You can translate a label into approximate memory cost with simple arithmetic. A 7 billion-parameter model at Q4_K_M is roughly 7 × 4.5 / 8 = about 3.9 GB. A 13 billion-parameter model at the same quantization is around 7.3 GB. A 70 billion-parameter model climbs above 39 GB. These figures cover the weights only; the active context window adds more on top.
Q4_K_M is the usual starting point because it is the smallest file that does not butcher quality. Drop to Q3_K_S or Q2_K only when nothing larger will fit. Stay at Q6_K or Q8_0 when you have memory to spare and care about subtle tasks like code review or long-form reasoning.
3. Match the model family to your task
Model size and quantization get you to “it runs.” The model family decides whether the answers are any good.
For everyday chat, rewriting, and summaries, the small open-weight chat-tuned families (Qwen, Gemma, Mistral, Llama) at 7B to 14B are the safest starting point. They are well represented on Ollama’s library and have been benchmarked across thousands of community use cases.
For code-heavy work, choose a model that has been instruction-tuned on code and explicitly tested on the languages you use. Pure chat models handle simple snippets fine but stumble on multi-file refactors. Our self-host a code completion setup walks through one concrete pair-up.
For tasks that need a long context window, like summarizing a long document or reasoning across several files, pick a model that advertises a long context in its model card, then test it with a real passage rather than a synthetic prompt. Local long-context support costs memory in ways that are easy to underestimate.
Specialist models (medical, legal, math) can be useful, but they trade breadth for narrow accuracy. Treat them as supplements, not replacements, until you have verified their behavior on your own material.
4. Run an honest test before you commit
A successful download only proves the model fits. Use three tests before you trust it:
- Your real task. Feed it one prompt from your actual work and judge the answer yourself. A model that explains trivia beautifully but fails your task is not useful.
- A grounding check. Give it a short passage and ask questions whose answers are present in that passage. Score its answers. Local models still invent facts, and small models invent more.
- A repeatability test. Run the same prompt twice. If the answer changes wildly, the model is not yet reliable for production work.
If the model passes these three for your task, you have found your default. If it does not, change one variable at a time. First swap quantization to one notch higher. Then try the next size up in the same family. Only then consider a different model family.
Keep an eye on system memory and GPU memory while you test. Both Ollama’s ollama ps and LM Studio’s developer panel show live usage. A setup that is silently swapping to disk will look slow and confused without obvious errors.
Bottom line
Choose the smallest open-weight model from a reputable family that fits comfortably in half your idle memory, downloaded at Q4_K_M or higher, and verified against your own work. Bump size or precision one step at a time only when the smaller setup fails a test you actually care about.
If you want more on the same theme, our Local AI coverage collects the running reviews and benchmarks.