What changed (2026-10-21): The blanket licence claim at the top of this guide was wrong and has been replaced. It said every model here was “permissive or commercially usable”; Gemma 3 is under Google’s custom Gemma terms and is gated: manual on Hugging Face, EXAONE 4.0 32B’s licence prohibits commercial use of the model, its derivatives and its output without a separate agreement, and Llama 4 Scout and Llama 3.3 70B carry Meta’s Llama community licences with their 700 million monthly active user clause. The Gemma 3 entries now name the gating. Measured Q4_K_M weight sizes against the Unsloth GGUF trees: Qwen 3 14B is 9.0 GB, not the “12GB memory footprint” the guide claimed, and Qwen 3 32B is 19.8 GB, not “about 22-24GB”. Qwen3-Coder-Next Q4_K_M confirmed at 48.5 GB, close enough to the original 48.5GB claim to keep. Google’s QAT announcement confirms Gemma 3 27B at 14.1 GB int4 against 54 GB BF16 and states it fits a single 24GB card; Gemma 3 4B is 2.6 GB int4 against 8 GB BF16. Deleted as unresourced rather than softened: the Gemma 3 27B Chatbot Arena Elo (1,338), EXAONE 4.0 32B’s Artificial Analysis scores (62 reasoning / 51 non-reasoning, versionless and no longer comparable after the index was rebased), its “trained on 14 trillion tokens” claim (not on the model card), and GLM-5’s “tops the intelligence index with 50.0”. GLM-5’s parameter counts were wrong: the card says 744B total and 40B active, not the 745B-A44B this guide gave. Qwen3-Coder-Next’s 71.3% SWE-bench Verified claim was dropped because its model card carries benchmark results only as images, not as a number that can be re-sourced. Held against primary sources and unchanged: Phi-4 14B at MMLU 84.8 and HumanEval 82.6, Ministral 3 14B at AIME ‘25 85%, gpt-oss-20b at Apache 2.0 running within 16GB via MXFP4, Qwen3.5-397B-A17B at LiveCodeBench 83.6 and AIME26 91.3, and Apple’s 273 GB/s memory bandwidth for the M4 Pro.
The open-weight model scene has moved fast over the past few months. OpenAI shipped its first open-source models. Alibaba dropped Qwen 3.5 with frontier-level performance. Zhipu AI released GLM-5, a 744B-parameter mixture-of-experts model. And Google’s quantization-aware training made 27B models fit on cards that used to struggle with 13B.
The question isn’t whether local AI is viable anymore. It’s which model to run on which hardware.
We broke down the current state of open-weight models into four tiers based on what you actually have sitting on your desk. Every recommendation accounts for quantization overhead, real-world token speeds, and practical usability - not just benchmark scores.
The Rules
Every model here is open-weight, but open-weight is not the same as an open licence. Most of the recommendations below are Apache 2.0 or MIT. Several are not. Gemma 3 runs under Google’s custom Gemma terms and is gated on Hugging Face, so the weights sit behind a manual access request even though Ollama pulls them without one. EXAONE 4.0 32B prohibits commercial use outright and requires a separate licence agreement for anything revenue-generating. Llama 4 Scout runs under the Llama 4 Community Licence, and Llama 3.3 70B (and DeepSeek-R1-Distill-Llama-70B, which is built on it) under the Llama 3.3 Community Licence; both carry a 700 million monthly active user clause. Check the licence before you build on any of them.
Throughput figures come from community benchmarks using Ollama, llama.cpp, or vLLM on the stated hardware, not from one controlled test rig, so treat them as the shape of the range rather than a promise. Quantization level is Q4_K_M unless noted otherwise: it is the standard balance between quality and memory savings. The Q4_K_M weight sizes quoted below are measured from the Unsloth GGUF builds rather than rounded to a round number, and they are weight sizes only, before KV cache and runtime overhead.
One critical thing to understand about running models locally: if the model doesn’t fit entirely in your GPU’s VRAM, layers get offloaded to system RAM. Each token then requires a round-trip across the PCIe bus. The result is roughly a 10x speed penalty. The line between “fits in VRAM” and “partially offloaded” is the single most important factor in local model performance.
Tier 1: 8GB VRAM (RTX 4060, RTX 3060 8GB, integrated graphics)
Your sweet spot: 7B–8B parameter models
This is the entry-level tier, and it’s better than you’d expect. An 8B model with Q4_K_M quantization needs about 6–7GB of VRAM, leaving just enough headroom for the KV cache during inference.
Top picks
Qwen 3 8B - The overall winner at this size. Strong reasoning, good multilingual support, and solid math performance. Runs at 40–53 tokens/second on an RTX 4060, which is faster than reading speed. The hybrid thinking mode lets it switch between quick responses and step-by-step reasoning depending on the prompt.
Gemma 3 4B (QAT) - Google’s quantization-aware trained version deserves special attention. Because it was trained with quantization in mind rather than quantized after the fact, the 4B model punches well above its weight. Google’s own figures put the int4 weights at 2.6 GB against 8 GB at BF16. If you’re extremely VRAM-constrained (think 6GB cards or integrated graphics), this is the one. Note that Gemma 3 is gated on Hugging Face, so accepting the Gemma terms there is a prerequisite for the weights; Ollama serves them without a separate request.
Phi-4 14B (at Q3 quantization) - Microsoft’s reasoning specialist. At Q3_K_M quantization it squeezes into 8GB VRAM, though you’re giving up some quality for the fit. Scores 84.8 on MMLU and 82.6 on HumanEval despite its size. If your use case is primarily math, logic, or code generation, the reasoning ability at this size is unmatched.
What to expect
General conversation, summarization, simple coding tasks, and writing assistance all work well at this tier. You’ll notice the models struggle with complex multi-step reasoning, long context (most 8B models cap at 8K–32K usable context), and tasks that require deep domain knowledge. But for a model running on a $300 GPU with zero data leaving your machine, it’s genuinely impressive.
Tier 2: 16GB VRAM (RTX 4060 Ti 16GB, RTX 4080, RTX A4000)
Your sweet spot: 14B–20B parameter models
The jump from 8B to 14B–20B is where local AI starts feeling like a real tool instead of a novelty. You get noticeably better coherence, stronger reasoning, and the ability to handle tasks that stump smaller models.
Top picks
GPT-OSS 20B - OpenAI’s first open-source model, and it’s a statement. Ships natively quantized in MXFP4 format, needing just 16GB of memory. Matches or exceeds OpenAI o3-mini on competition math, health queries, and general reasoning benchmarks. Runs at around 140 tokens/second on an RTX 4080. Apache 2.0 license. The fact that an OpenAI model now runs on consumer hardware under an open license would have been unthinkable a year ago.
Qwen 3 14B - The best general-purpose model at this tier. Delivers 62 tokens/second with a 9.0 GB (8.4 GiB) Q4_K_M weight footprint (9,001,753,984 bytes), leaving room for context and parallel workloads. Hybrid reasoning mode, strong coding ability, and excellent instruction following. It’s the default recommendation for most people with 16GB cards.
Ministral 3 14B - Mistral’s edge-optimized model. Hits 70 tokens/second while fitting entirely in 16GB VRAM. The reasoning variant scores 85% on AIME ‘25, which is competition math territory. All under Apache 2.0. Particularly strong at producing concise, well-structured output - it tends to generate fewer tokens than comparable models while maintaining quality, which translates to faster effective task completion.
What to expect
At this tier, you can handle most day-to-day AI tasks without feeling limited. Code generation and review works well. Long-form writing is coherent. Simple agentic tasks (tool calling, multi-step workflows) become viable. Context windows push to 128K tokens on some models, though actual usable context is typically lower before quality degrades.
Tier 3: 24GB VRAM (RTX 4090, RTX 3090, RTX A5000)
Your sweet spot: 27B–32B parameter models (or quantized 70B)
This is the tier where local AI gets serious. The RTX 4090 remains the consumer king - 24GB of GDDR6X with 1.01 TB/s bandwidth. It’s enough to run 32B models at full Q4 quantization with room to spare, or push into 70B territory with aggressive quantization.
Top picks
Qwen 3 32B - The best all-around model on a 24GB card. The Q4_K_M weights measure 19.8 GB, or 18.4 GiB (19,762,150,048 bytes), so a 24GB card (24 GiB) holds them with roughly 5.6 GiB left for KV cache and runtime. Performance at this size represents a clear step up from the 14B class across every benchmark. If you want one model that handles everything - coding, writing, analysis, math, conversation - this is it.
EXAONE 4.0 32B - LG AI Research’s hybrid reasoning model, which ships both a non-reasoning and a reasoning mode and carries 131,072 tokens of context. Its earlier Artificial Analysis Intelligence Index scores are gone from this page: the index has been rebased since publication, so the versionless figures this guide originally quoted no longer mean anything comparable. The licence is the part worth acting on. EXAONE 4.0 32B is published under the EXAONE licence, which prohibits commercial use of the model, its derivatives and its output without a separate commercial licence agreement from LG AI Research. That is stricter than a usage cap you can grow into; if you are building a product, this one needs a licence conversation before you ship.
Gemma 3 27B (QAT) - Google’s quantization-aware trained version drops from 54 GB at BF16 to just 14.1 GB at int4, per Google’s own QAT announcement. That is unusually efficient, and Google states the int4 build fits on a single 24GB card. Strong vision capabilities too, if you need image understanding. An earlier version of this guide quoted a Chatbot Arena Elo for this model; that figure has been removed because Arena leaderboards turn over models continuously and it could not be re-sourced to a current listing. Like the 4B, Gemma 3 27B is gated on Hugging Face under Google’s Gemma terms.
DeepSeek R1 Distill 32B - If reasoning is your priority, this distilled version of DeepSeek’s R1 retains most of the full model’s step-by-step problem-solving ability. Particularly good at math, code, and structured analysis. The trade-off is that it’s slower at simple tasks because it defaults to verbose reasoning chains.
Quantized 70B models - The RTX 4090 can handle Llama 3.3 70B or Qwen 2.5 72B at Q3 or Q4 quantization, hitting about 26–52 tokens/second depending on the model and quantization level. You lose some quality from the heavy quantization, but a 70B model - even quantized - outperforms a 32B model at full precision on most tasks. Worth trying if response quality matters more than speed.
What to expect
At this tier, local models start competing with cloud API responses. Complex reasoning chains work. Multi-turn conversations maintain coherence over long exchanges. Agentic coding tasks - where the model needs to plan, execute, and self-correct - become practical. The main remaining gap with frontier cloud models is in extremely long contexts and bleeding-edge tool use.
Tier 4: Apple Silicon (M-series Macs with 32GB–128GB unified memory)
Your sweet spot: depends entirely on unified memory
Apple Silicon rewrites the rules because unified memory means the GPU and CPU share the same memory pool with no copy penalty. An M4 Max with 128GB unified memory can load models that would need multiple NVIDIA GPUs.
The catch: memory bandwidth. LLM inference is almost entirely bandwidth-bound during token generation, and Apple Silicon bandwidth varies dramatically by chip. An M3 Max (400 GB/s) actually generates tokens faster than an M4 Pro (273 GB/s) despite being the older chip. So pay attention to bandwidth, not just total memory.
Top picks by memory tier
32GB (M4 Pro base): Qwen 3 14B or GPT-OSS 20B at Q4 quantization. Expect 25–35 tokens/second. The same models that shine on 16GB NVIDIA cards work well here, with more headroom for context.
64GB (M4 Pro/Max): Qwen 3 32B at Q4–Q5 is the top recommendation, running at 12–18 tokens/second. Qwen3-Coder-Next 80B (3B active parameters) also fits at Q4_K_M - its 48.5GB quantized weight size squeaks in under 64GB with room for overhead. This is a coding-specialized model that beats DeepSeek V3.2 on SWE-Bench while using 12x less compute per token.
96–128GB (M4 Max/Ultra): This is where things get wild. Llama 3.3 70B runs at Q4–Q5 quantization at 8–15 tokens/second. Qwen 3.5 397B-A17B - a frontier-class model with only 17B active parameters - fits at 3-bit quantization on 192GB or 4-bit on 256GB, generating around 25+ tokens/second with MoE offloading. You’re running a model that competes with GPT-5.2 on your laptop.
MLX matters
If you’re on Apple Silicon, use the MLX backend rather than the standard GGUF/llama.cpp path. MLX is built specifically for Apple’s hardware and consistently outperforms the generic path by 20–40% on token generation speed.
The MoE Wild Card
Mixture of Experts models have changed what’s possible at every tier. The key insight: a 400B MoE model with 17B active parameters needs to store all 400B parameters in memory, but only computes with 17B per token. So inference speed is similar to a 17B dense model, but you need the memory of a 400B model.
This makes MoE models perfect for high-memory, bandwidth-rich setups (read: Apple Silicon Macs). The standouts:
- Qwen 3.5 397B-A17B (February 2026) - Frontier performance. 83.6 on LiveCodeBench, 91.3 on AIME26. Needs ~214GB at Q4 via Unsloth’s dynamic quantization.
- GLM-5 (February 2026) - 744B total parameters with 40B active per token, confirmed on the model card, which also records a pretraining scale-up from GLM-4.5’s 355B/32B on 28.5T tokens. Needs server hardware or a massive Mac. Its former “tops the intelligence index” claim has been removed: the Artificial Analysis index has since been rebased and GLM-5 has been succeeded on that board.
- Llama 4 Scout 109B-A17B - 10 million token context window. Strong multimodal capabilities. Fits on a 64GB Mac at Q4.
- Qwen3-Coder-Next 80B-A3B - 80B total with 3B activated per token, 10 of 512 experts active, and 262,144 tokens of native context. The Q4_K_M build measures 48.5 GB (48,528,320,544 bytes), which is what lets it sit on a 64GB Mac. The best coding model you can run locally on consumer hardware, period.
Quick Reference: Best Model Per Hardware
| Hardware | Best General | Best Coding | Best Reasoning | Speed |
|---|---|---|---|---|
| 8GB VRAM | Qwen 3 8B | Qwen 3 8B | Phi-4 14B (Q3) | 40–53 t/s |
| 16GB VRAM | GPT-OSS 20B | GPT-OSS 20B | Ministral 3 14B | 60–140 t/s |
| 24GB VRAM | Qwen 3 32B | Qwen 3 32B | EXAONE 4.0 32B (non-commercial licence) | 30-52 t/s |
| Mac 64GB | Qwen 3 32B | Qwen3-Coder-Next | Qwen 3 32B | 12–18 t/s |
| Mac 128GB | Llama 3.3 70B | Qwen3-Coder-Next | DeepSeek-R1-Distill-Llama-70B | 8-15 t/s |
Two names deserve a note. “DeepSeek R1 70B” in earlier versions of this table was imprecise: DeepSeek-R1 itself is a 671B mixture-of-experts model, and the 70B model is the Llama-3.3-70B distill of it, published as DeepSeek-R1-Distill-Llama-70B. And EXAONE 4.0 32B holds the reasoning slot at 24GB only because it has no clear substitute at that size; its licence rules out commercial use, so for anything you intend to ship, treat Qwen 3 32B as the pick.
How to Get Started
All of these models are available through Ollama with a single command:
# Install Ollama (macOS/Linux)
curl -fsSL https://ollama.com/install.sh | sh
# Pull and run any model
ollama run qwen3:8b # 8GB tier
ollama run gpt-oss:20b # 16GB tier
ollama run qwen3:32b # 24GB tierFor Apple Silicon users wanting the MLX backend, LM Studio provides a clean interface with automatic MLX optimization. For power users who want more control, llama.cpp and vLLM offer the best performance tuning options.
If you want a full ChatGPT-style web interface running locally, check out our guide to setting up Ollama with Open WebUI - published earlier today.
The Bottom Line
October 2026 is a better time yet to run AI locally than this guide first argued in February. OpenAI ships open-weight models under Apache 2.0. A $300 GPU runs models that beat GPT-3.5. A $1,600 Mac Mini handles models that rival GPT-4. And a maxed-out Mac Studio runs frontier-class 400B parameter models from your desk.
The performance gap between local and cloud is shrinking every month. For many tasks - writing, coding, analysis, research - it’s already closed. The remaining advantages of cloud models are in bleeding-edge reasoning, extremely long contexts, and complex multi-modal tasks. For everything else, you can keep your data on your own machine and still get excellent results.
Your hardware decides your tier. This guide tells you which model to pick once you know your tier. Beyond that, the only step left is ollama pull.