Local AI by VRAM Tier - 8GB | 12GB | 16GB | 24GB | 32GB
Deep dives: Chat | Coding | Translation | Vision | Speech | Agents
August 2026 refresh: this page was a full model generation behind the rest of the series. Chat, coding, vision and agents are rebuilt around the current Qwen generation, coding gains Kwaipilot’s KAT-Coder-V2.5-Dev, and vision now has two defensible picks rather than one stale one.
24GB is where local AI stops being a compromise. A 27B dense model fits with context to spare, a document-grade vision model fits at a quantisation that is worth running, and a full voice pipeline plus a chat model can share the card. This is the enthusiast tier, and this year it finally has models built for it rather than models squeezed into it.
Updated August 2026. Refreshed monthly as models and runtimes change.
Your Hardware
24GB VRAM cards: RTX 3090 (used market) and RTX 4090. The 4090 is the faster card for inference; the 3090 is the cheaper route to the same capacity. Prices on both move constantly, so check current listings.
How to read the VRAM numbers on this page. Every figure is the static size of the model weights at the named quantisation, taken from published GGUF file sizes, with two exceptions that are labelled where they appear: the Whisper figures are OpenAI’s own required-VRAM numbers rather than weight sizes, and the Step-Audio-EditX figures are StepFun’s stated GPU memory rather than weight sizes. The KV cache is allocated on top and grows with context length, so each number is a floor, not a total. Anything within 1-2GB of the 24GB ceiling fits with minimal context room, and we flag those.
The rough math: at Q4_K_M, 24GB holds the 27-35B class with several gigabytes for context. At Q6_K a 27B still fits, but the margin gets thin. Mixture-of-experts models do not help with capacity, only speed: all experts stay resident even though only a few activate per token.
Quick Reference
| Use Case | Best Pick | Static weights | Quantisation | Note |
|---|---|---|---|---|
| Chat | Qwen3.6-27B | see chat guide | Q4-class | Newest open-weight Qwen dense, Apache-2.0 |
| Chat (measured) | Qwen3.5-27B | 16.74 GB | Q4_K_M | Published GGUF sizes, generous context room |
| Coding (autocomplete) | Qwen2.5-Coder 32B | ~20 GB | Q4_K_M | FIM, 128K context |
| Coding (agentic) | Qwen3.6-27B | see coding guide | Q4-class | 77.2% SWE-bench Verified, self-reported |
| Coding (new MoE) | KAT-Coder-V2.5-Dev | not published | - | Apache-2.0, 35B total / 3B active |
| Translation | TranslateGemma 27B | 17 GB | Ollama build | Custom Gemma terms, not a blanket open licence |
| Vision (quality) | Qwen3.5-27B + mmproj | 23.38 GB | Q6_K + mmproj-F16 | Minimal context room |
| Vision (speed) | Qwen3.5-35B-A3B + mmproj | 22.92 GB | Q4_K_M + mmproj-F16 | MoE, open llama.cpp CUDA bug |
| Speech | Parakeet + Step-Audio-EditX + 8B LLM | ~20 GB | mixed | Apache-2.0 TTS |
| Agents | GLM-4.7-Flash | 19 GB | Q4_K_M | MIT licence |
Chat & General Assistant
The dense pick is Qwen3.6-27B, the newest open-weight model in the Qwen line and Apache-2.0. The Qwen3.6 repository ships only two sizes, 27B and 35B-A3B, and both are listed on Ollama. We found no open-weight Qwen3.7 or 3.8 release, which is an observation about what is published, not a claim about Alibaba’s plans.
For figures you can size a card against, the previous generation is the one with published GGUF blob sizes. Qwen3.5-27B is 14.98GB at IQ4_XS, 16.74GB at Q4_K_M, 22.45GB at Q6_K and 28.60GB at Q8_0. On a 24GB card that means Q4_K_M with real context headroom, Q6_K with a thin margin, and Q8_0 not at all. Both generations are Apache-2.0 and ungated.
This replaces the Qwen 3 30B MoE and Qwen 3 32B picks that stood here, along with a throughput claim from an aggregator we no longer cite. The chat deep dive adds Gemma 4 26B-A4B as the mixture-of-experts efficiency option at this tier.
Full comparison: local chat model guide.
Coding
Autocomplete: Qwen2.5-Coder 32B at roughly 20GB, with fill-in-the-middle and 128K context. Table 16 of the Qwen team’s own technical report reports 92.7% HumanEval for it. It is an older generation than everything else here and still the best local inline-completion model, because the newer general-purpose releases are not FIM-trained.
Agentic: Qwen3.6-27B, which the coding deep dive tracks at 77.2% on SWE-bench Verified. That figure is self-reported by the lab, as are essentially all SWE-bench numbers for open-weight models.
New this month: Kwaipilot KAT-Coder-V2.5-Dev, released in late July 2026 (the card does not state the exact day). It is a 35B-total / 3B-active mixture-of-experts model, Apache-2.0, with 262K context, and the text-only variant of KAT-Coder-V2.5, so serving it through vLLM or SGLang needs the --language-model-only flag because the parent carries vision. Hugging Face lists more than 50 community quantisations of it for llama.cpp, LM Studio, Jan and Ollama; the model card itself does not mention quantised builds.
Its self-reported scores: SWE-bench Verified 69.40%, SWE-bench Multilingual 63.00%, SWE-bench Pro 45.96%, Terminal-Bench 2.1 41.02%. Two warnings. They are the lab’s own numbers, not an independent evaluation. And SWE-bench Verified and SWE-bench Pro are different benchmarks over different task sets, so reading 69.40 against 45.96 as a decline is a mistake. Compare Verified against Verified only.
Kwaipilot publishes no VRAM figure for it and we are not going to invent one. It is the same size class as Qwen3.6-35B-A3B, so plan for a mixture-of-experts footprint and measure it on your own card.
Permissive alternative: Devstral Small 2 from Mistral, a 24B dense Apache-2.0 model with a long context window, covered with its benchmark numbers in the coding deep dive.
Full comparison: local coding model guide.
Translation
TranslateGemma 27B is listed on Ollama at 17GB, leaving room to translate a whole document in one pass and keep terminology consistent. Its technical report puts it at MetricX 3.09 and COMET22 84.4, figures published by the model’s own authors. The licence is Google’s custom Gemma terms with a prohibited-use policy attached, not a blanket open licence.
For low-resource languages, Aya Expanse 32B at 19.80GB in a Q4_K_M community GGUF is the strongest local option, with two constraints: it is CC-BY-NC-4.0, so non-commercial use only, and the weights are access-gated on Hugging Face. For commercial work, use TranslateGemma, or a current Qwen model when the output needs to read naturally rather than mechanically.
Full comparison: local translation model guide.
Vision
This section previously recommended Qwen2.5-VL 32B on the strength of a DocVQA score. Both halves of that are wrong now. The model is two generations old, and DocVQA and ChartQA under those exact names appear in neither Qwen family’s own reporting, so that number did not come from the vendor. Alibaba reports OmniDocBench 1.5, CC-OCR, CharXiv and AI2D instead.
Qwen3.5 supersedes both the Qwen3 text line and the Qwen3-VL vision line, and every Qwen3.5 dense size is vision-capable. That gives 24GB owners two defensible picks.
Quality pick: Qwen3.5-27B at Q6_K. Weights are 22.45GB and the mmproj-F16 projector adds 0.93GB, for 23.38GB. That fits a 24GB card with minimal context room, so it suits short-prompt document work rather than long multi-image sessions. Drop to Q4_K_M plus mmproj at 17.67GB when you need context, or IQ4_XS at 14.98GB to keep a second model resident.
MoE speed pick: Qwen3.5-35B-A3B at Q4_K_M. Weights are 22.02GB plus 0.90GB of projector, for 22.92GB, also minimal context room. Only about 3B parameters activate per token, so throughput beats the dense 27B on batch OCR work.
Read this before choosing the MoE. llama.cpp issue #25717, opened 15 July 2026, is still open: a CUDA illegal memory access on vision requests with Qwen3.5-35B-A3B plus mmproj, involving flash attention. A parallel vLLM issue reporting a CUDA illegal memory access in the GDN kernel is also open. If your workflow is CUDA plus vision plus flash attention on this model size, the dense 27B is the safer choice today.
The supporting benchmarks, comparing Qwen3.5-27B against the far larger Qwen3-VL-235B-A22B, are self-reported by Alibaba across its own two product lines. There is no independent leaderboard to check them against: OpenCompass OpenVLM is JavaScript-only and returned no data, and Artificial Analysis runs no vision benchmark for either family.
| Benchmark | Qwen3.5-27B | Qwen3-VL-235B-A22B |
|---|---|---|
| MMMU | 82.3 | 80.6 |
| MMMU-Pro | 75.0 | 69.3 |
| MathVision | 86.0 | 74.6 |
| MathVista (mini) | 87.8 | 85.8 |
| RealWorldQA | 83.7 | 81.3 |
| MMStar | 81.0 | 78.7 |
| OCRBench | 89.4 | 87.5 |
| OmniDocBench 1.5 | 88.9 | 84.5 |
| CC-OCR | 81.0 | 81.5 |
Note the last row. On CC-OCR the 27B loses to the 235B model, 81.0 against 81.5. It is the only loss among the nine rows above, though Alibaba’s full table carries three more: SimpleVQA (56.0 against 61.3), RefCOCO average (90.9 against 91.1) and ODInW13 (41.1 against 43.2). If your workload is that kind of character-level OCR, CC-OCR is the row that matters most.
Newest option at this size: Qwen3.6-27B is Apache-2.0, vision-capable and the same parameter count, with self-reported MMMU 82.9 against Qwen3.5-27B’s 82.3, RealWorldQA 84.1 against 83.7, and OCRBench tied at 89.4. The gains are small enough that we are not rebuilding the tier around it, but it is the version to try first if it runs cleanly for you.
Two asymmetries worth knowing. Alibaba publishes official GGUF conversions for Qwen3-VL but not for Qwen3.5, so every Qwen3.5 GGUF is third-party work such as the Unsloth builds. If a first-party GGUF matters more to you than the newer generation, Qwen3-VL-32B is 19.76GB at Q4_K_M plus a 1.20GB projector, 20.96GB together. Runtime support for Qwen3.5 is real rather than announced: llama.cpp PR #19468 merged in February 2026 with dense and MoE support verified against Qwen Team preview checkpoints in both vision and text-only modes.
Full comparison: local vision model guide.
Speech
24GB runs a complete voice assistant with room left over. Transcription: Canary-Qwen-2.5B for English, Parakeet TDT 0.6B v3 at roughly 2GB for the smallest footprint, or Whisper large-v3 at roughly 10GB for broad language coverage. We are not restating a ranking between them this month: the models are individually unchanged, but the live Open ASR leaderboard order could not be reliably reconfirmed in this pass. If you run whisper.cpp, update to v1.9.2 (4 August 2026); v1.9.0 added native NVIDIA Parakeet support, so one runtime now covers both families.
Text-to-speech, with a correction. Our July write-up had the top two open-weight models the wrong way round. On the Artificial Analysis open-weights TTS arena pulled on 6 August 2026, Fish Audio S2 Pro leads at 1121 Elo and Step-Audio-EditX is second at 1109.
Step-Audio-EditX is still our recommendation, and the reason is the licence, not the score. Fish Audio S2 Pro ships under a research licence that is non-commercial; Step-Audio-EditX is Apache-2.0. An Elo gap of about 12 points does not survive contact with a licence that forbids the use you probably have in mind.
Best combination here: Parakeet (~2GB) plus Step-Audio-EditX (~12GB) plus an 8B-class chat model (~6GB) is roughly 20GB together. Speak a question, get an answer read back in a cloned voice, entirely locally.
Full comparison: local speech model guide.
Agents
GLM-4.7-Flash is the 24GB agent pick: a 30B-total / 3B-active mixture-of-experts model under an MIT licence, the most permissive terms on this page. The MoE design matters for agents specifically, because every tool call is a round trip and an eight-step workflow means eight or more inference passes.
When the priority is clean, parseable function calls rather than long autonomous planning, Qwen3.5-27B and Qwen3.5-35B-A3B are the reliable choices and both fit with context headroom. The agent deep dive carries the function-calling numbers for all three, with the warning that they come from different benchmarks and are not directly comparable.
Full comparison: local agent model guide.
Getting Started
# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh
# The essentials for 24GB
ollama pull qwen3.6:27b # Chat + agentic coding
ollama pull qwen2.5-coder:32b # Autocomplete (~20GB)
ollama pull translategemma:27b # Translation (17GB)
# These do not all fit at once; Ollama loads and unloads as needed
ollama run qwen3.6:27bModel tags shift as libraries update, so check the Ollama library for current ones. For vision, note that no first-party Qwen3.5 GGUF exists, so you are pulling a community conversion whichever runtime you use.
The 24GB Advantage: Real Quantisation Choices
At the lower tiers the question is whether a model fits. At 24GB it is which quantisation you run it at:
- Qwen3.5-27B at Q4_K_M (16.74GB) leaves around 7GB for KV cache, which is a lot of context
- The same model at Q6_K plus vision (23.38GB) trades nearly all of that context for output quality
- A vision model plus a speech pipeline can share the card instead of taking turns
- Autocomplete and an agentic coder both stay warm, so switching modes costs no reload
Every total above is static weights. Leave headroom on whichever model does the long-context work.
When to Upgrade
24GB covers nearly everything a single consumer card can do. Moving to 32GB buys higher quantisation on the same 27-35B models, comfortable context on a vision model at Q6, and more room for multi-model pipelines.
What it does not buy is the frontier. The open-weight models topping the mid-2026 leaderboards run to hundreds of billions of parameters, and even the smallest of the current crop lands around 82.5GB as a 1-bit community quantisation, still beyond any single consumer GPU. Those models are for servers.
See our 32GB guide for the frontier consumer tier.
All eleven local-AI guides, by hardware tier and by task, are indexed at Local AI by VRAM: Which Models Fit Your GPU.