Local AI by VRAM Tier - 8GB | 12GB | 16GB | 24GB | 32GB
Deep dives: Chat | Coding | Translation | Vision | Speech | Agents
August 2026 refresh: this was the last page in the series still recommending the previous model generation. Chat, coding, vision and agents are rebuilt on the current picks, coding gains Kwaipilot’s KAT-Coder-V2.5-Dev, the top two open-weight text-to-speech models were listed in the wrong order, and every GPU-throughput and street-price figure that traced to aggregator blog posts has been removed rather than reattributed.
32GB is the top of the consumer stack, and what it buys is not a class of model the smaller tiers cannot touch. It is margin. The same 27B to 35B models that fit a 24GB card with a gigabyte to spare run here at a higher quantisation, with a vision projector attached, or alongside a second model that stays warm.
Updated August 2026. Refreshed monthly as models and runtimes change.
Your Hardware
32GB of VRAM on a single consumer card means the RTX 5090, which is the only card in that class. Availability and pricing on it move constantly, so check current listings rather than a number written months ago.
How to read the VRAM numbers on this page. Every figure is the static size of the model weights at the named quantisation, taken from published GGUF file sizes, with two exceptions that are labelled where they appear: the Whisper figures are OpenAI’s own required-VRAM numbers rather than weight sizes, and the Step-Audio-EditX figures are StepFun’s stated GPU memory rather than weight sizes. The KV cache is allocated on top and grows with context length, so each number is a floor, not a total. Anything within 1-2GB of the 32GB ceiling fits with minimal context room, and that is flagged where it happens.
The rough math: at Q4_K_M, 32GB holds the 27-35B class with several gigabytes free for context or a second resident model. At Q6_K a 27B fits with real headroom. At Q8_0 a 27B fits with minimal context room. Mixture-of-experts models do not help with capacity, only speed: all experts stay resident even though only a few activate per token.
No tokens-per-second figures appear on this page. Earlier versions quoted an average inference throughput and a percentage speedup over the previous card generation, both taken from GPU-benchmark blog posts this site no longer cites. No primary measurement replaced them, so those claims are gone rather than reattributed to something else. Throughput depends on the runtime, the quantisation, the context length and the batch size, so measure it on your own card.
Quick Reference
| Use Case | Best Pick | Static weights | Quantisation | Note |
|---|---|---|---|---|
| Chat | GLM-4.7-Flash | 19 GB | Q4_K_M | MIT licence, Q8_0 is 32 GB and leaves no context room |
| Chat (Qwen) | Qwen3.6-27B | see chat guide | Q4-class | Newest open-weight Qwen dense, Apache-2.0 |
| Chat (measured) | Qwen3.5-27B | 28.60 GB | Q8_0 | Published GGUF size, minimal context room |
| Coding (autocomplete) | Qwen2.5-Coder 32B | 27 GB | Q6_K | FIM, 128K context |
| Coding (agentic) | Qwen3.6-27B | see coding guide | Q4-class | 77.2% SWE-bench Verified, self-reported |
| Coding (new MoE) | KAT-Coder-V2.5-Dev | not published | - | Apache-2.0, 35B total / 3B active |
| Translation | TranslateGemma 27B + Qwen3.5-9B | 17 + 5.68 GB | Ollama build + Q4_K_M | Custom Gemma terms on the 27B |
| Vision (default) | Qwen3.5-27B + mmproj | 23.38 GB | Q6_K + mmproj-F16 | Real context headroom at this tier |
| Vision (max precision) | Qwen3.5-27B + mmproj | 29.53 GB | Q8_0 + mmproj-F16 | Minimal context room |
| Vision (speed) | Qwen3.5-35B-A3B + mmproj | 22.92 GB | Q4_K_M + mmproj-F16 | MoE, open llama.cpp CUDA bug |
| Speech | Canary-Qwen + Step-Audio-EditX + 14B-class LLM | ~28 GB | mixed | Apache-2.0 TTS |
| Agents | GLM-4.7-Flash | 19 GB | Q4_K_M | MIT licence |
Chat & General Assistant
The chat pick at this tier is GLM-4.7-Flash, a 30B-total / 3B-active mixture-of-experts model under an MIT licence, the most permissive terms on this page. It already runs on a 24GB card at Q4_K_M, 19GB on Ollama; what 32GB adds is a much larger KV cache at that quantisation, or the 32GB Q8_0 build with almost none. Full precision is not on the table at this tier, because the BF16 build is roughly 60GB. Its reasoning scores are self-reported on its own model card and are carried, with that label, in the chat deep dive.
The Qwen pick is Qwen3.6-27B, the newest open-weight model in the Qwen line and Apache-2.0. The Qwen3.6 repository ships only two sizes, 27B and 35B-A3B, and both are listed on Ollama. No open-weight Qwen3.7 or 3.8 release was found, which is an observation about what is published, not a claim about Alibaba’s plans. Qwen3.6-35B-A3B is the mixture-of-experts sibling and sits fully resident here with context room left over.
For figures you can size a card against, the previous generation is the one with published GGUF blob sizes. Qwen3.5-27B is 16.74GB at Q4_K_M, 22.45GB at Q6_K and 28.60GB at Q8_0. On a 32GB card that means Q8_0 fits with minimal context room, Q6_K fits with roughly 9GB of headroom, and Q4_K_M leaves enough space to keep a second model loaded. Qwen3.5-35B-A3B is 22.02GB at Q4_K_M and 28.85GB at Q6_K; its Q8_0 build is 36.90GB and does not fit this card. Both generations are Apache-2.0 and ungated.
This replaces the Qwen 3 30B MoE and EXAONE 4.0 32B picks that stood here. The EXAONE recommendation and its MMLU-Redux figure have been dropped rather than re-sourced: no primary source for either was confirmed in this pass, and the 24GB page dropped the same claim for the same reason. A throughput claim from an aggregator page went with them.
Full comparison: local chat model guide.
Coding
Autocomplete: Qwen2.5-Coder 32B at Q6_K, 27GB, which is the configuration a 24GB card cannot hold. Table 16 of the Qwen team’s own technical report reports 92.7% HumanEval for the model. It is an older generation than everything else on this page and still the best local inline-completion model, because the newer general-purpose releases are not trained for fill-in-the-middle. The same model at Q4_K_M is roughly 20GB, which is the build to pull when something else has to share the card.
Agentic: Qwen3.6-27B, which the coding deep dive tracks at 77.2% on SWE-bench Verified. That figure is self-reported by the lab, as are essentially all SWE-bench numbers for open-weight models. At this tier it runs at a higher quantisation, or with a very large context, rather than at the minimum that fits.
New this month: Kwaipilot KAT-Coder-V2.5-Dev, released in late July 2026 (the card does not state the exact day). It is a 35B-total / 3B-active mixture-of-experts model, Apache-2.0, with 262K context, and the text-only variant of KAT-Coder-V2.5, so serving it through vLLM or SGLang needs the --language-model-only flag because the parent carries vision. Hugging Face lists more than 50 community quantisations of it for llama.cpp, LM Studio, Jan and Ollama; the model card itself does not mention quantised builds.
Its self-reported scores: SWE-bench Verified 69.40%, SWE-bench Multilingual 63.00%, SWE-bench Pro 45.96%, Terminal-Bench 2.1 41.02%. Two warnings. They are the lab’s own numbers, not an independent evaluation. And SWE-bench Verified and SWE-bench Pro are different benchmarks over different task sets, so reading 69.40 against 45.96 as a decline is a mistake. Compare Verified against Verified only.
Kwaipilot publishes no VRAM figure for it, and none is invented here. It is the same size class as Qwen3.6-35B-A3B, so plan for a mixture-of-experts footprint and measure it on your own card. What this tier offers it is the option of a higher quantisation than a 16GB card allows.
Permissive alternative: Devstral Small 2 from Mistral, a 24B dense Apache-2.0 model with a long context window, which fits comfortably at Q8_0, 25.06GB. One thing to check before building on it: Mistral marks this model deprecated as of 27 February 2026 and points new integrations at a hosted model rather than an open-weight replacement. The Apache-2.0 weights stay downloadable and keep working, but nobody is maintaining them.
Dual setup: Qwen2.5-Coder 32B at Q4_K_M (roughly 20GB) for autocomplete plus Qwen3.5-9B at Q4_K_M (5.68GB) for chat is about 26GB of weights together, so both stay warm and switching modes costs no reload.
Full comparison: local coding model guide.
Translation
TranslateGemma 27B is listed on Ollama at 17GB, and at this tier it no longer has to take the card to itself. TranslateGemma 27B plus Qwen3.5-9B at Q4_K_M (5.68GB) is about 22.7GB of weights combined, with real context headroom left: route literal and technical pairs to the dedicated translator and prose to the general model. Its technical report puts it at MetricX 3.09 and COMET22 84.4, figures published by the model’s own authors. The licence is Google’s custom Gemma terms with a prohibited-use policy attached, not a blanket open licence.
Ollama also publishes higher-precision TranslateGemma 27B builds alongside the 17GB default: a q8_0 tag at 30GB, which fits this card but takes it to itself with almost no context room, and a bf16 tag at 55GB that does not fit at all. The Q6_K TranslateGemma figure that used to appear on this page has no primary source behind it and has been removed.
Single-model maximum: Qwen3.5-27B at Q6_K, 22.45GB, Apache-2.0, and the widest claimed language list of anything that fits one card.
Full comparison: local translation model guide.
Vision
This section previously recommended Qwen2.5-VL 32B on the strength of a DocVQA score. Both halves of that are wrong now. The model is two generations old, and DocVQA and ChartQA under those exact names appear in neither Qwen family’s own reporting, so that number did not come from the vendor. Alibaba reports OmniDocBench 1.5, CC-OCR, CharXiv and AI2D instead.
Qwen3.5 supersedes both the Qwen3 text line and the Qwen3-VL vision line, and every Qwen3.5 dense size is vision-capable. Unlike the 16GB tier, where neither family fits without a lossy quantisation and this series makes no recommendation at all, 32GB has genuine headroom, and the question is only how much of it to spend on precision.
Every figure below is static weights at the named quantisation and excludes the KV cache, which on a vision model grows with image resolution as well as context length.
| Option | Quantisation | Weights + mmproj | Free before KV cache |
|---|---|---|---|
| Qwen3.5-27B | Q6_K + mmproj-F16 | 22.45 + 0.93 = 23.38 GB | roughly 8.6 GB |
| Qwen3.5-27B | Q8_0 + mmproj-F16 | 28.60 + 0.93 = 29.53 GB | minimal context room |
| Qwen3.5-35B-A3B | Q4_K_M + mmproj-F16 | 22.02 + 0.90 = 22.92 GB | roughly 9 GB |
The working default is Q6_K. The same 23.38GB configuration on a 24GB card leaves under a gigabyte before the KV cache, which is the difference between a demo and a pipeline. Here it leaves roughly 8.6GB, which is enough for long documents and high-resolution images.
Q8_0 is a hedge, not a measured gain. No benchmark comparison between Q6_K and Q8_0 builds of this model was found, so treat the extra precision as insurance against quantisation artifacts on dense small text rather than as a known improvement, and note that 29.53GB leaves minimal context room even on this card.
MoE speed pick: Qwen3.5-35B-A3B at Q4_K_M, 22.92GB with the projector. Only about 3B parameters activate per token, so throughput beats the dense 27B on batch OCR work. Read this before choosing it: llama.cpp issue #25717, opened 15 July 2026, is still open and reports a CUDA illegal memory access on vision requests with Qwen3.5-35B-A3B plus mmproj, involving flash attention. A parallel vLLM issue reporting a CUDA illegal memory access in the GDN kernel is also open. If your workflow is CUDA plus vision plus flash attention at this model size, the dense 27B is the safer choice today. A Q8_0 build of the MoE model does exist, but at 36.90GB it does not fit this card; Q6_K at 28.85GB is the highest-precision option that does.
The supporting benchmarks, comparing Qwen3.5-27B against the far larger Qwen3-VL-235B-A22B, are self-reported by Alibaba across its own two product lines. There is no independent leaderboard to check them against: OpenCompass OpenVLM is JavaScript-only and returned no data, and Artificial Analysis runs no vision benchmark for either family.
| Benchmark | Qwen3.5-27B | Qwen3-VL-235B-A22B |
|---|---|---|
| MMMU | 82.3 | 80.6 |
| MMMU-Pro | 75.0 | 69.3 |
| MathVision | 86.0 | 74.6 |
| MathVista (mini) | 87.8 | 85.8 |
| RealWorldQA | 83.7 | 81.3 |
| MMStar | 81.0 | 78.7 |
| OCRBench | 89.4 | 87.5 |
| OmniDocBench 1.5 | 88.9 | 84.5 |
| CC-OCR | 81.0 | 81.5 |
Note the last row. On CC-OCR the 27B loses to the 235B model, 81.0 against 81.5. It is the only loss among the nine rows above, though Alibaba’s full table carries three more: SimpleVQA (56.0 against 61.3), RefCOCO average (90.9 against 91.1) and ODInW13 (41.1 against 43.2). If your workload is that kind of character-level OCR, CC-OCR is the row that matters most.
Newest option at this size: Qwen3.6-27B is Apache-2.0, vision-capable and the same parameter count, with self-reported MMMU 82.9 against Qwen3.5-27B’s 82.3, RealWorldQA 84.1 against 83.7, and OCRBench tied at 89.4. No GGUF weight sizes for Qwen3.6 were collected in this pass, so no VRAM figure is quoted for it. The gains are small enough that the tier is not rebuilt around it, but it is the version to try first if it runs cleanly.
Two asymmetries to know about. Alibaba publishes official GGUF conversions for Qwen3-VL but not for Qwen3.5, so every Qwen3.5 GGUF is third-party work such as the Unsloth builds. If a first-party GGUF matters more to you than the newer generation, Qwen3-VL-32B is 19.76GB at Q4_K_M plus a 1.20GB projector, 20.96GB together, and its Q8_0 build at 34.82GB does not fit this card at all. Runtime support for Qwen3.5 is real rather than announced: llama.cpp PR #19468 merged in February 2026 with dense and MoE support verified against Qwen Team preview checkpoints in both vision and text-only modes.
Multi-model stack. 32GB also holds vision, chat and speech at once. Qwen3.5-9B at Q4_K_M plus its 0.92GB projector is 6.60GB, which leaves most of the card for a chat model and a speech pair.
Full comparison: local vision model guide.
Speech
32GB runs a full-quality voice pipeline with a capable conversational model behind it. Transcription: Canary-Qwen-2.5B at 5.12GB for English, Parakeet TDT 0.6B v3 at roughly 2GB for the smallest footprint, or Whisper large-v3 at roughly 10GB for broad language coverage. No accuracy ranking between them is restated this month: the models are individually unchanged, but the live Open ASR leaderboard order could not be reliably reconfirmed in this pass, so the ranking and the word-error-rate comparison that used to appear across this series have been removed rather than repeated. If you run whisper.cpp, update to v1.9.2 (4 August 2026); v1.9.0 added native NVIDIA Parakeet support, so one runtime now covers both families.
Text-to-speech, with a correction. The July write-up had the top two open-weight models the wrong way round. On the Artificial Analysis open-weights TTS board pulled on 6 August 2026, Fish Audio S2 Pro leads at 1121 Elo and Step-Audio-EditX is second at 1109.
Step-Audio-EditX is still the recommendation, and the reason is the licence rather than the ranking. Fish Audio S2 Pro ships under the fish-audio-research-license, which is non-commercial; Step-Audio-EditX is Apache-2.0. An Elo gap of about 12 points does not survive contact with a licence that forbids the use most self-hosters have in mind.
Best combination here: Canary-Qwen (5.12GB) plus Step-Audio-EditX (~12GB) plus a 14B-class chat model (~11GB) is roughly 28GB together. Transcription, expressive cloned speech and a capable conversational model, all resident on one card and nothing leaving the machine.
Full comparison: local speech model guide.
Agents
GLM-4.7-Flash is the 32GB agent pick: the same 30B-total / 3B-active mixture-of-experts model as the chat section, under an MIT licence, with enough KV-cache headroom here to hold a long tool history without eviction. The mixture-of-experts design matters for agents specifically, because every tool call is a round trip and a ten-step workflow means ten or more inference passes.
When the priority is clean, parseable function calls rather than long autonomous planning, Qwen3.6-27B is the dense alternative and runs at this tier with a higher quantisation or a very large context. The agent deep dive carries the function-calling numbers for both, with the warning that they come from different benchmarks and are not directly comparable.
Full comparison: local agent model guide.
Getting Started
# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh
# The essentials for 32GB
ollama pull glm-4.7-flash # Chat + agents
ollama pull qwen3.6:27b # Dense chat + agentic coding
ollama pull qwen2.5-coder:32b # Autocomplete
ollama pull translategemma:27b # Translation (17GB)
# These do not all fit at once; Ollama loads and unloads as needed
ollama run glm-4.7-flashModel tags shift as libraries update, so check the Ollama library for current ones. For vision, the qwen3.5 library page carries tags with “Text, Image” capabilities, but note that no first-party Qwen3.5 GGUF exists, so you are pulling a community conversion whichever runtime you use.
The 32GB Advantage: Margin, Not New Categories
Nothing on this page is a model class the 24GB tier cannot touch. What changes is how much room is left after the weights load:
- Qwen3.5-27B at Q6_K plus its vision projector (23.38GB) leaves roughly 8.6GB here against under a gigabyte on a 24GB card
- Qwen2.5-Coder 32B at Q6_K (27GB) replaces the Q4 build, which is where hallucinated function names and wrong API calls start dropping off
- A dedicated translator and a general model stay resident together at about 22.7GB instead of taking turns
- A full voice pipeline plus a 14B-class model fits at roughly 28GB
- KAT-Coder-V2.5-Dev can run at a higher quantisation than a 16GB card allows, though its footprint is not published and has to be measured
Every total above is static weights at the named quantisation. Leave headroom on whichever model does the long-context work.
What 32GB Does Not Buy
It does not buy the frontier. The open-weight models topping the mid-2026 leaderboards run to hundreds of billions of parameters: even the smallest current build of DeepSeek V4, the V4-Flash-0731 beta, lands around 82.5GB as a 1-bit community quantisation, and the larger releases run from hundreds of gigabytes into the terabytes. Those are server models, and the chat guide covers them with their licences and runtime status.
It also does not buy a large jump in quality over 24GB. The model list is nearly identical between the two tiers; what differs is the quantisation you run it at and how many models stay warm at once. If a 24GB card is already in the machine, the honest answer is that it covers most of this page with tighter margins. The case for 32GB is document-grade vision at Q6 with real context, near-lossless autocomplete, and multi-model pipelines that never have to swap.
All eleven local-AI guides, by hardware tier and by task, are indexed at Local AI by VRAM: Which Models Fit Your GPU.