The open-weight LLM landscape has a marketing problem. Every week brings announcements of 700-billion-parameter models that “match GPT-4.5” - but require a server room to run. If you have an RTX 4090 or a MacBook Pro, you’re not running GLM-5’s 744B parameters. Period.
So let’s skip the flagship hype and focus on what actually works on consumer hardware. Here are the open-weight models delivering real performance in 2026, tested on the GPUs you actually own.
The Reality Check: VRAM Tiers
Before diving into models, here’s the brutal math. The VRAM you have determines what you can run:
| VRAM | What Fits (Q4 quantization) | Practical Context |
|---|---|---|
| 8GB | 7-8B models | Limited context |
| 12GB | 8-14B models | Comfortable context |
| 16GB | 14-27B models | Good context headroom |
| 24GB | 27-35B models | Full context window |
Q4 quantization cuts memory substantially with limited quality loss, though “4-bit” is a label rather than a bit width: in llama.cpp’s own reference table the common Q4_K_M build stores about 4.89 bits per weight, and Q8_0 about 8.50. Expect roughly a 3x saving against a 16-bit model, not a clean 4x. If someone tells you their model “runs on consumer hardware” without mentioning quantization, they’re probably hiding something.
The 14B Sweet Spot: Where the Action Is
If you have a modern GPU with 12-16GB VRAM, the 14B parameter tier is where you get the best quality-per-VRAM. Here’s how the leaders stack up:
Qwen 3 14B: The All-Rounder
Qwen’s 14B model consistently outperforms or matches models twice its size on math and reasoning benchmarks. Its Q4_K_M weights measure 9.0 GB, or 8.4 GiB (9,001,753,984 bytes), leaving room for context even on a 16GB card.
Real-world performance: 60-70 tokens/second on RTX 4090, around 40 tok/s on RTX 4080.
Best for: General assistant tasks, coding, reasoning. If you want one model that does everything competently, this is it.
DeepSeek-R1 14B Distill: The Reasoning Specialist
If you specifically need chain-of-thought reasoning - working through problems step-by-step rather than blurting answers - the DeepSeek R1 distilled variant is purpose-built for this.
Real-world performance: 58.6 tokens/second on RTX 4090 with Ollama, Q4 quantized (9GB model), in Database Mart’s tests.
Best for: Math, logic puzzles, code debugging where you want to see the thinking. The reasoning traces are genuinely useful, not just filler.
Phi-4 14B: The STEM Specialist
Microsoft’s Phi-4 punches well above its weight on technical problems. In Microsoft’s own technical report it scores 80.4% on MATH (competition-level problems), ahead of GPT-4o’s 74.6%. On code generation it reports 82.6% HumanEval, above Llama-3.3-70B (78.9%) and Qwen 2.5 72B (80.4%) in the same table. These are vendor-reported figures.
Real-world performance: around 62 tokens/second on an RTX 4080 (16GB) and 69 on an RTX 4090 at Q4_K_M, per the awesomeagents.ai leaderboard - slightly faster than DeepSeek-R1 14B on the same cards.
Best for: STEM homework, mathematical proofs, scientific reasoning. If your use case is technical, Phi-4 is worth a look.
The Budget Tier: 8B Models That Don’t Suck
Not everyone has 16GB VRAM. Here’s what works on older or mid-range hardware:
Qwen 3 8B
Dominates this tier on reasoning and math. At Q4_K_M, it runs on mid-range 6-8GB cards at 40+ tokens/second. The quality gap from 14B is real, but it’s still better than most cloud chatbots from two years ago.
Llama 3.1 8B
If coding is your primary use case, Llama’s 72.6% HumanEval score is hard to beat at this size. It’s also the most battle-tested option - the community has optimized inference for this model more than any other.
The honest take: 8B models are usable, not magical. They’ll struggle with complex multi-step reasoning but handle everyday tasks fine.
The Stretch Goal: 27-35B Models on 24GB
If you have an RTX 4090 or M3 Max with 48GB+ unified memory, you can reach the next tier:
Qwen3.5-35B-A3B (35B MoE)
This mixture-of-experts architecture only activates a fraction of parameters per token: 35B parameters in total, with 3B active per token. It carries a Q4_K_M build of 22.0 GB, or 20.5 GiB (22,016,023,168 bytes), so it fits on 24GB VRAM with a little room left over for context. Published under Apache 2.0.
Real-world performance: 45-60 tokens/second on RTX 4090. On a 256GB M3 Ultra with MoE offloading, you can hit 25+ tok/s.
Best for: Users who need strong general quality without the cloud. The MoE architecture means you are not paying the full compute cost of 35B parameters per token.
An earlier version of this section called this model “Qwen 3.5 Medium”. That name does not match any published model repository; on Hugging Face the 35B MoE model is published as Qwen3.5-35B-A3B (on Ollama, qwen3.5:35b-a3b), and that is the name to search for.
Gemma 3 27B
Google’s entry fits on a single 24GB GPU and delivers solid general-purpose performance. However, community benchmarks show it runs surprisingly slow compared to similar-sized models - around 50 tok/s on RTX 5090, notably slower than alternatives.
The honest take: Gemma 3 27B is capable but not the performance leader. If speed matters, Qwen3.5-35B-A3B is the better pick at this tier.
What About the Flagships?
Let’s address the elephant: GLM-5 (744B), Llama 4 Maverick (400B), and other headline-grabbers. Can you run them locally?
Short answer: No, not usefully.
- GLM-5 has 744B parameters (40B active); at FP8 (one byte per parameter) the weights alone are roughly 744 GB, and Z.ai’s own vLLM and SGLang recipes shard it across eight GPUs
- Llama 4 Scout (109B total) can squeeze onto a 24GB GPU at 1.78-bit quantization via Unsloth (a 33.8GB file), at ~20 tok/s and at an extreme level of compression
- Mistral Large 3 is a cluster model: even with Mistral’s compressed NVFP4 checkpoint, Mistral describes running it on a single 8x A100 or 8x H100 node
These models are open-weight, which is great for transparency and fine-tuning. But “open-weight” doesn’t mean “runs on your gaming rig.”
The Practical Setup
If you’re starting from zero, here’s what actually works:
Software: Ollama for simplicity, or vLLM if you want maximum performance and don’t mind configuration.
First model to try: Qwen 3 14B at Q4_K_M quantization. It’s the safest all-purpose pick.
Check your VRAM first:
# Linux
nvidia-smi
# macOS
system_profiler SPDisplaysDataType | grep "VRAM\|Chipset"Install and run:
ollama pull qwen3:14b-q4_K_M
ollama run qwen3:14b-q4_K_MWhat This Means
The gap between “what’s announced” and “what’s practical” has never been wider. Flagship open-weight models now require datacenter hardware, while consumer-friendly models have gotten genuinely good.
The 14B tier in 2026 delivers what took 70B+ parameters two years ago. If you have a decent GPU and want to run AI locally - for privacy, for cost savings, for control - the practical options are better than ever.
Just ignore the 700B parameter announcements. They’re not for you. The 14B models are, and they’re getting the job done.
What changed (2026-10-21): The biggest correction is a model name. This guide called the 35B mixture-of-experts model “Qwen 3.5 Medium”, which is not the name of any published repository; the model is published as Qwen3.5-35B-A3B, 35B total with 3B active per token under Apache 2.0. The name now matches the repo everywhere it appeared, including the “honest take” line under Gemma 3 27B. Its Q4_K_M build measures 22.0 GB, which confirms the guide’s claim that it fits on a 24GB card. Qwen 3 14B’s footprint was stated as “about 10.7GB VRAM”; the actual Q4_K_M weights are 9.0 GB (9,001,753,984 bytes), so that is corrected. The “Q4 reduces memory by roughly 4x” line was wrong in the way this site has been wrong about before: 4-bit is a label, not a bit width. In llama.cpp’s reference table Q4_K_M stores about 4.89 bits per weight and Q8_0 about 8.50, so the saving against a 16-bit model is roughly 3x, and the text now says that. The Mistral Large 3 line said the model “needs” an 8x A100 or 8x H100 node; Mistral’s own announcement presents the NVFP4 checkpoint as letting you run it on a single 8-GPU node, so the wording now matches the source. Re-verified and unchanged: GLM-5 at 744B total and 40B active, with roughly 744 GB of weights at one byte per parameter; Llama 4 Scout at 109B total and 33.8 GB for the 1.78-bit build at about 20 tokens per second on a 24GB card; Phi-4 14B’s MATH 80.4 against GPT-4o’s 74.6, and its HumanEval 82.6 against Llama-3.3-70B’s 78.9 and Qwen 2.5 72B’s 80.4; Llama 3.1 8B’s HumanEval 72.6. Throughput numbers sourced to community leaderboards are left as they were and are still attributed as such.