Local AI by VRAM Tier - 8GB | 12GB | 16GB | 24GB | 32GB
Deep dives: Chat | Coding | Translation | Vision | Speech | Agents
A local chatbot that runs on your own hardware, answers your questions without sending data to anyone, and costs nothing per query. That’s the pitch. The reality depends entirely on your GPU.
Updated August 2026. Refreshed monthly as models and runtimes change.
What changed this month: the giants section now covers Kimi K3, MiniMax-M3 and Inkling alongside GLM-5.2 and DeepSeek V4, and every citation has been moved onto model cards, licence files and official runtime listings. The tier picks themselves are unchanged from July: Qwen3.5-9B at 8-12GB, GPT-OSS 20B at 16GB, Qwen3.6-27B at 24GB, GLM-4.7-Flash at 32GB.
This guide tracks the major open-weight chat models across five VRAM tiers. Every model here runs through Ollama at Q4_K_M quantization unless noted otherwise.
How the Comparison Works
The signals that matter for chat:
- Artificial Analysis Intelligence Index - a blended cross-benchmark score, useful for ranking general capability
- GPQA / AIME - graduate-science and competition-math questions, the ceiling tests for reasoning
- VRAM at Q4 - whether it actually fits your card with room for context
- License - whether you can use it commercially (it varies more than people expect)
Two grounding notes. Every VRAM figure below is the static weight size at the named quantization and excludes the KV cache, which grows with context length, so anything within a gigabyte or two of a tier boundary should be read as “fits with minimal context room”. And every benchmark score published by the lab that trained the model is labelled self-reported, because none of these numbers come from an independent re-run.
8GB VRAM {#8gb}
GPUs: RTX 4060, RTX 3060 8GB, RTX 3070
The entry tier. You’re limited to models around 4-9B parameters, or a small MoE, at Q4. In 2026 that still means genuinely useful conversations.
Candidates
| Model | Params | VRAM (weights) | License | Note |
|---|---|---|---|---|
| Qwen3.5-9B | 9B | 5.68 GB (Q4_K_M) | Apache 2.0 | Ungated, best all-round fit |
| Gemma 4 E4B | ~4B effective | per Ollama tag list | Apache 2.0 | 128K context |
| LFM2.5-8B-A1B | 8.3B / 1.5B active (MoE) | under ~6 GB | LFM Open License v1.0, commercial use capped at $10M revenue | Built for tool calling, 128K context |
Winner: Qwen3.5-9B
Qwen3.5-9B remains the best all-round 8GB chat model. Its Q4_K_M weights are 5.68GB, which leaves an 8GB card enough room for a normal conversation once the KV cache is accounted for, and the repository is Apache-2.0 and ungated. The Qwen 3.6 generation (below) is stronger, but its smallest members are 27B and up, so they do not fit here; the 3.5 9B is still the right pick at this tier.
Efficiency pick: Gemma 4 E4B. Google’s Gemma 4 uses an effective-parameter design, and the E4B build is the smallest entry in Ollama’s Gemma 4 listing with 128K context. Read the current tag size off that page rather than trusting a round number. Note the licence change: unlike Gemma 3n, which shipped under Google’s custom gated Gemma terms, the Gemma 4 line is published under Apache 2.0 and is ungated.
Tool-calling pick: LFM2.5-8B-A1B. Liquid AI’s on-device mixture-of-experts activates only ~1.5B of 8.3B parameters per token, stays under ~6GB, and was built to emit function calls by default - the pick if your 8GB machine needs an agent rather than a chatbot.
The honest take: 8GB chat models handle everyday questions, brainstorming, and simple tasks well. They struggle with long multi-step reasoning and deep world knowledge. Ask one to plan detailed logistics and you’ll see the limits.
For all use cases at this level, see the 8GB VRAM complete guide.
12GB VRAM {#12gb}
GPUs: RTX 3060 12GB, RTX 4070
The comfortable tier. You can run a 9B with lots of context headroom, or reach into 15B dense reasoning models.
Winner: Qwen3.5-9B (with headroom), or Apriel-1.6-15B for reasoning
Running Qwen3.5-9B on 12GB gives you plenty of room for long context on top of its 5.68GB of Q4_K_M weights. When you want stronger step-by-step reasoning, step up to Apriel-1.6-15B-Thinker - a 15B dense model under an MIT license with 131K context and a self-reported score of 57 on version 3.0 of the Artificial Analysis Intelligence Index. ServiceNow’s own GGUF is 8.79GB at Q4_K_M, so it fits a 12GB card with roughly 3GB of context room.
Gemma 4 also ships a 12B dense model listed in Ollama’s library; its exact Q4 weight size is best read from the tag list there rather than from a round number, and it carries the same Apache 2.0 licence.
For all use cases at this level, see the 12GB VRAM complete guide.
16GB VRAM {#16gb}
GPUs: RTX 4060 Ti 16GB, RTX 5060, Intel Arc A770, AMD RX 7800 XT
The sweet spot. 15-20B models fit comfortably, and small MoEs become practical.
Candidates
| Model | Params | VRAM (weights) | License | Key figure |
|---|---|---|---|---|
| GPT-OSS 20B | 20B (MoE) | ~14 GB (Ollama default tag) | Apache 2.0 | AA Intelligence Index 15 (v4.1) |
| Apriel-1.6-15B-Thinker | 15B dense | 8.79 GB (Q4_K_M) | MIT | AA Intelligence Index 57 (v3.0, per its card) |
| Qwen3.6-35B-A3B | 35B / 3B active (MoE) | per Ollama tag list | Apache 2.0 | Newer generation, tight fit |
Winner: GPT-OSS 20B
OpenAI’s open-weight model is still the strongest all-rounder at this tier. gpt-oss-20b is Apache-2.0, lands at 14GB in Ollama’s default build (check the current tag size before pulling), and scores 15 on version 4.1 of the Artificial Analysis Intelligence Index, against a median of 9 for open-weight models of similar size. The 52.1 quoted here previously was measured on an earlier version of that index, which has since been rebased. It now has real competition, but nothing at this tier clearly beats it as an all-rounder yet.
Reasoning alternative: Apriel-1.6-15B-Thinker. It is MIT-licensed, and at 8.79GB it leaves comfortable context room on a 16GB card. Its card claims 57 on the Artificial Analysis Intelligence Index, but that figure is measured on index v3.0 while the GPT-OSS number above is v4.1, so the two cannot be placed side by side and no ordering between them is claimed here. Try it if GPT-OSS feels shallow on hard reasoning.
New generation, if it fits: Qwen3.6-35B-A3B is an Apache-2.0 mixture-of-experts model from the current Qwen line. A 35B-class MoE keeps all experts resident, so it does not fit a 16GB card at Q4; expect an aggressive quant with modest context, and read the exact tag size from Ollama’s Qwen3.6 listing before committing. Treat it as a “fits at aggressive quant” option, not a comfortable default.
For all use cases at this level, see the 16GB VRAM complete guide.
24GB VRAM {#24gb}
GPUs: RTX 3090, RTX 4090
This is where local models start competing with cloud APIs on quality. The 27-31B tier fits on a single consumer GPU.
Candidates
| Model | Params | VRAM (weights) | License | Note |
|---|---|---|---|---|
| Qwen3.6-27B | 27B dense | ~17 GB (Q4) | Apache 2.0 | New default at this tier |
| Gemma 4 26B-A4B | 26B / 4B active (MoE) | per Ollama tag list | Apache 2.0 | ~4B active parameters per token |
| Gemma 4 31B | 31B dense | per Ollama tag list | Apache 2.0 | Tight at 24GB, comfortable at 32GB |
Winner: Qwen3.6-27B
The current-generation dense model is the new 24GB default. Qwen3.6 is Apache-2.0, and a 27B dense model at Q4 is around 17GB of weights (the same-size Qwen3.5-27B Q4_K_M build is 16.74GB), leaving real headroom for KV cache on a 24GB card. This is the model that makes “local can replace my chat subscription for most tasks” a defensible claim.
MoE efficiency: Gemma 4 26B-A4B activates only ~4B parameters per token, which is what makes it the smoothest high-context option in the Gemma line at this tier. Sizes are listed per tag in Ollama’s Gemma 4 listing, and the Apache 2.0 licence applies across the Gemma 4 line.
Dense alternative: Gemma 4 31B is the largest dense Gemma 4. It is a tight fit at 24GB with limited context and more comfortable on a 32GB card.
For all use cases at this level, see the 24GB VRAM complete guide.
32GB VRAM {#32gb}
GPUs: RTX 5090
The frontier. 32GB gives comfortable high-context 27-35B models and MoE headroom.
Candidates
| Model | Params | VRAM | License | Key figure |
|---|---|---|---|---|
| GLM-4.7-Flash | 30B / 3B active (MoE) | 19 GB (Q4_K_M), 32 GB (Q8_0) | MIT | AIME 2025 91.6, GPQA 75.2 (self-reported) |
| Qwen3.6-35B-A3B | 35B / 3B active (MoE) | ~22 GB (Q4) | Apache 2.0 | Full in-VRAM, long context |
| Qwen3.6-27B | 27B dense | ~17 GB (Q4) | Apache 2.0 | Max quant / max context |
Winner: GLM-4.7-Flash
GLM-4.7-Flash is a MIT-licensed 30B-A3B mixture-of-experts model. Its Q4_K_M build is 19GB, which fits a 24GB card with room for context; the Q8_0 build is 32GB, which takes a 32GB card to itself. True full precision is the BF16 build at roughly 60GB and does not fit any card in this guide. Its reasoning scores are the standout at this tier: AIME 2025 at 91.6 and GPQA at 75.2, both self-reported on its model card. Unsloth’s guide covers the exact quant options.
Same-tier alternative: Qwen3.6-35B-A3B fully resident gives 32GB owners generous context headroom while staying inside the well-supported Qwen tooling. Budget around 22GB of weights at Q4 plus KV cache; the same-size Qwen3.5-35B-A3B Q4_K_M build is 22.02GB.
For all use cases at this level, see the 32GB VRAM complete guide.
The Open Weights That Do Not Fit a Consumer GPU
Several of the largest open-weight releases of mid-2026 are worth knowing about, but none of them run on a single 8-32GB card, and most do not run on a single machine at all. Sizes below are community GGUF build sizes across quantization levels. Every benchmark figure in this section is vendor self-reported.
Kimi K3 is a 2.8T-parameter mixture-of-experts model with 104B active parameters, on Hugging Face, where the repository was created 2026-06-13 and last updated 2026-07-27. Community GGUF builds run from 594GB at 1-bit up to 1.56TB. vLLM and SGLang had support from day zero. The fact that matters most for readers of this guide is a runtime one: llama.cpp support has not been merged. PR #26185 is still open, and because Ollama and LM Studio both sit on llama.cpp, neither of them can run Kimi K3 either. Its custom “Kimi K3 License” permits commercial use, but $20M/yr in revenue triggers a required separate agreement, and more than 100M monthly active users or $20M/mo in revenue requires UI branding.
MiniMax-M3 is roughly 428B total with about 23B active, natively handling text, image and video. GGUF builds range from 128GB to 852GB. llama.cpp merged both the sparse-attention PR (#24908) and the vision PR (#25113) on 2026-07-26, with CUDA confirmed and ROCm, Vulkan and Metal untested. vLLM lists M3 among its supported models, but correctness fixes were still landing there in early August 2026, so treat that path as young. The model card publishes no benchmark table, so no score is quoted here; an SWE-bench figure that appeared in an earlier version of this page could not be traced to any MiniMax source. Its “MiniMax Community License” allows commercial use with a “Built with MiniMax M3” attribution and the same $20M/yr revenue gate.
Inkling, from Thinking Machines Lab, is 975B total with 41B active and native text, image and audio input. It is Apache-2.0, the most permissive licence in this group, and was published on Hugging Face on 2026-07-14. SGLang and vLLM recipes exist; llama.cpp and Ollama support is unverified.
GLM-5.2 is 753B (GLM-5 is 744B with 40B active, GLM-5.1 is 754B), released under MIT with no additional restrictions. GGUF builds run 217GB to 1.51TB, and Ollama carries glm-5.2 and glm-5.1 tags. Two benchmark figures circulate for this family and they are frequently confused: GLM-5.2 self-reports 62.1 on SWE-bench Pro, while GLM-5 self-reports 77.8 on SWE-bench Verified. Those are different benchmarks with different task sets and difficulty, so the two numbers cannot be compared with each other and should never be placed side by side.
DeepSeek V4 ships as V4-Pro (1.6T total, 49B active) and V4-Flash (284B total, 13B active), plus a V4-Flash-0731 beta dated 2026-07-31. The licence is MIT, confirmed by reading the LICENSE file in the repository. Even the 1-bit V4-Flash-0731 GGUF is 82.5GB, rising to 162GB at higher precision, so it is out of reach of any single consumer GPU at any quantization. DeepSeek self-reports SWE-bench Verified up to 80.6%, but that figure is specific to the “V4-Pro-Max” reasoning-effort mode; across effort modes the reported range is 73.6% to 80.6%, so the top number should never be quoted on its own.
This guide sticks to what a single 8-32GB GPU can actually run, which is why none of the five appear in the tier tables above.
Cross-Tier Summary
| Tier | Best Pick | Reasoning Alt | Efficiency Pick |
|---|---|---|---|
| 8GB | Qwen3.5-9B | LFM2.5-8B-A1B (tools) | Gemma 4 E4B |
| 12GB | Qwen3.5-9B | Apriel-1.6-15B-Thinker | Gemma 4 E4B |
| 16GB | GPT-OSS 20B | Apriel-1.6-15B-Thinker | Gemma 4 12B |
| 24GB | Qwen3.6-27B | - | Gemma 4 26B-A4B (MoE) |
| 32GB | GLM-4.7-Flash | Qwen3.6-35B-A3B | Qwen3.6-27B |
Quick Start
Every model listed above runs through Ollama. Install it, pull a model, and start chatting:
# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh
# Pick your model (choose based on your VRAM tier above)
ollama pull qwen3.5:9b # 8-12GB tier
ollama pull gpt-oss:20b # 16GB tier
ollama pull qwen3.6:27b # 24GB tier
ollama pull glm-4.7-flash # 32GB tier
# Start chatting
ollama run qwen3.5:9bModel tags change as libraries update - check the Ollama library for the exact current tag. For a ChatGPT-style web interface, pair Ollama with Open WebUI; the full setup guide walks through it.
What These Models Can’t Do
Local chat models have real limits compared to frontier cloud models like Claude, GPT, or Gemini:
- Long, complex instructions - multi-step tasks with many constraints still fray below the 27B tier
- Very recent knowledge - these models were trained months ago; they don’t know about last week
- Creative writing at scale - short-form is fine, novel-length coherence degrades
- Reliable factual accuracy - they hallucinate, especially on obscure topics. Always verify claims
The 27-35B models at this tier come closest to closing the gap. But for now, local chat is best as a private, instant, free alternative for everyday conversations - not a wholesale replacement for frontier models on the hardest tasks.
All eleven local-AI guides, by hardware tier and by task, are indexed at Local AI by VRAM: Which Models Fit Your GPU.