A chat model that lives on your card answers without sending anything to a server, costs nothing per query, and removes the recurring subscription - if the model actually fits, runs at a useful speed, and is licensed for your use. This page is a VRAM-tier chooser for the open-weight chat families that currently satisfy those three tests on a single consumer GPU: Qwen3.5, Qwen3.6, Gemma 4, and gpt-oss. Every figure below was pulled from the Hugging Face model API, the model’s own card, or the live Ollama tag page on 2026-10-17.
How the comparison works
Four signals matter for chat picks:
- VRAM at Q4_K_M. the static weight size at the named Ollama tag; excludes the KV cache, which grows with context length. Anything within a gigabyte of a tier boundary is “fits with minimal context room”, not “fits with comfort”.
- Native context. the maximum context length the model can serve without an extension scheme. Qwen3.5 and Qwen3.6 publish 262,144 tokens natively, extensible to 1,010,000 via YaRN; Gemma 4 ships 128K (E2B / E4B) or 256K (12B / 26B / 31B); gpt-oss ships 128K.
- License. Apache 2.0 for every model named below, with two notes. Gemma 4 ships under the standard Apache 2.0 text (Gemma 4 license), replacing the prior Gemma Terms of Use that gated Gemma 3. Llama 3.2 stays on the Llama 3.2 community license with a 700M monthly-active-user clause; it appears only as a low-VRAM fallback.
- Architecture. dense vs MoE. Dense models activate every parameter per token and pay a flat VRAM cost. MoE models activate only a slice - gpt-oss-20b is 21B total with 3.6B active; GLM-4.7-Flash is 30B total with 3B active - which is why they fit at sizes their total parameter count would seem to forbid.
Every benchmark score in this page is taken from the model’s own card. Independent re-runs were not performed.
8 GB and below
GPUs: RTX 4060, RTX 3060 8 GB, RTX 3070, Apple Silicon base configs.
| Model | Params | Ollama tag (Q4) | Native context | License |
|---|---|---|---|---|
| Qwen3.5-0.8B | 0.8B | qwen3.5:0.8b 1.2-1.3 GB | 256K | Apache 2.0 |
| Qwen3.5-2B | 2B | qwen3.5:2b 2.7-3.1 GB | 256K | Apache 2.0 |
| Llama 3.2 1B | 1B | llama3.2:1b 1.3 GB | 128K | Llama 3.2 community |
| Llama 3.2 3B | 3B | llama3.2:3b 2.0 GB | 128K | Llama 3.2 community |
Pick. qwen3.5:4b (3.3-4.0 GB, 256K context, Apache 2.0) is the smallest Qwen 3.5 that is genuinely useful for chat rather than autocomplete. The full 9B variant is too large at the default Ollama tag (6.6-7.6 GB); on a strict 8 GB card with no headroom for context, the 4B is the right choice. Below that, the 2B and 0.8B Qwen 3.5 sizes hold 256K context but the chat quality is closer to autocomplete than to assistant. Llama 3.2 1B and 3B remain the smallest viable chat on Apple Silicon or older Nvidia cards where Qwen 3.5 cannot be built, but they ship under the Llama 3.2 community license with its 700M MAU clause.
The local AI without a GPU page covers the same picks via Ollama on integrated graphics; a tuned Qwen3.5-4B is the practical ceiling for CPU-only chat.
12 GB tier
GPUs: RTX 3060 12 GB, RTX 4070, Apple Silicon M-class with 12-16 GB unified memory.
Pick. qwen3.5:9b (6.6-7.6 GB default, 262,144 native context extensible to 1,010,000 via YaRN, Apache 2.0) is the strongest chat on this card. Its model card lists MMLU-Pro 82.5, GPQA Diamond 81.7, and IFEval 91.5, which is the strongest verifiable score at this VRAM. Gemma 4 E4B (6.6-9.5 GB on the Ollama tag page, 128K context, Apache 2.0) is the multimodal alternative if you also want image input - Gemma 4 E4B has a vision encoder in addition to text. Gemma 4 E2B (4.6-7.5 GB) is the smaller of the two and a stronger pick if you want to leave headroom for context on an 8 GB card.
The 12 GB card is also the right place to try the Qwen 3.5 9B at higher quant (the default tag is Q4_K_M); the 9B BF16 build is 17.92 GB and does not fit. Stick with the default tag unless a specific quant is named in the Ollama library.
16 GB tier
GPUs: RTX 4060 Ti 16 GB, RTX 5060, Intel Arc A770 16 GB, AMD RX 7800 XT.
Pick. gpt-oss:20b (14 GB default tag, 128K context, Apache 2.0). OpenAI’s open-weight model is 21B total with 3.6B active per token, stored in MXFP4 on the Ollama build - which is why the 14 GB tag fits a 16 GB card. It is the strongest verifiable all-rounder at this VRAM tier, and its HF card ships as Apache-2.0 and ungated. Two siblings matter at this tier: qwen3.5:9b (6.6-7.6 GB, 256K context, Apache 2.0) for the cheaper half of the card and a longer context window; gemma4:12b (7.7-8.0 GB, 256K context, Apache 2.0, multimodal) for an image-capable chat at a similar footprint.
The 16 GB tier is also where the 120B gpt-oss first becomes worth thinking about. gpt-oss:120b is 65 GB, which does not fit on a single 16 GB card at any quantization. Multi-GPU serving is covered in the when one GPU is not enough page.
24 GB tier
GPUs: RTX 3090, RTX 4090, Apple Silicon M-class with 24-48 GB unified memory.
Pick. qwen3.6:27b (18-19 GB default, 262,144 native context extensible to 1,010,000 via YaRN, Apache 2.0). The Qwen3.6-27B model card describes a 27B dense model with the same hybrid attention layout as Qwen3.5 - Gated DeltaNet layers interleaved with Gated Attention layers - but at the current-generation size. This is the card on which “local can replace my chat subscription for most tasks” becomes a defensible claim. Gemma 4 26B (16-19 GB, 256K context, Apache 2.0, multimodal) is the closest competitor at the same VRAM; Gemma 4 31B (19-20 GB, 256K context, Apache 2.0, multimodal) is a tighter fit at 24 GB and a comfortable fit at 32 GB.
32 GB tier
GPUs: RTX 5090, RTX 4090 with extra headroom, dual-card 2x16 GB configurations treated as one logical card.
Pick. qwen3.6:35b (23-24 GB default, 256K context, Apache 2.0) - the 35B-A3B mixture-of-experts from Qwen, served under the qwen3.6:35b Ollama tag. The GLM-4.7-Flash model card is a near-twin in footprint (30B/3B MoE, MIT, 131K context) and worth pulling if you specifically want MIT over Apache 2.0. Both ship in Q4_K_M and fit a 32 GB card with meaningful context headroom. The dense ceiling at this tier is qwen3.6:27b at higher quantization - the same model as the 24 GB pick, run with more headroom for the KV cache rather than a heavier quant.
For cluster-serving and multi-GPU configurations that go beyond a single 32 GB card, the team-serving guide covers the network plumbing; the local AI inference speed page covers what to expect when context grows.
What local chat cannot do
Local chat models have real limits compared to frontier hosted models. Even the 32 GB tier holds back from the front of cloud on:
- Very recent knowledge. these models were trained on data that closed months ago; they do not know about last week
- Reliable factual accuracy. all chat models hallucinate, especially on obscure topics; always verify
- Novel-length coherence. short-form creative writing is fine, multi-chapter-generation drifts
- Hard multi-step reasoning. reasoning-tuned models (covered in the best reasoning models page) close some of the gap, but the ceiling below 32 GB is real
For privacy-sensitive chat - tax documents, medical questions, anything that should not leave the device - the local path is the only path; for everyday chat on a 24 GB card, qwen3.6:27b is the strongest verifiable pick.
The bottom line
Across all five VRAM tiers the answer converges on a small set of names: qwen3.5:9b for 8-12 GB, gpt-oss:20b for 16 GB, qwen3.6:27b for 24 GB, and either qwen3.6:35b (Apache 2.0, MoE 35B-A3B) or GLM-4.7-Flash (MIT, MoE 30B/3B) for 32 GB. Every model named in this page is Apache 2.0 (Qwen3.5, Qwen3.6, Gemma 4, gpt-oss) except GLM-4.7-Flash, which is MIT, and they can be deployed commercially without a separate agreement; the only license exception is Llama 3.2, kept here as a low-VRAM fallback with its 700M MAU clause intact. Every tag size quoted is the live Ollama tag as of 2026-10-17 and will drift; confirm on the Ollama library before committing to a deployment.