Pick a quant and the next question is what you actually gave up. The most repeated rule of thumb on the local-AI internet, “Q4 loses about 1% quality” and “Q5 is nearly lossless”, is repeated everywhere and sourced almost nowhere. The closest the original toolchain comes is naming the measures - perplexity and KL Divergence - and then not publishing either.
This page is the answer that is actually sourced: published KL Divergence and MMLU numbers across quant levels for a real 27B model, where the cliff is, and how to measure the cost on your own workload. If you have not picked a format yet, start with GGUF vs AWQ vs GPTQ vs MLX; for VRAM tier, the GPU hub is the index.
What the damage looks like in real numbers
Unsloth publishes a per-quant KL Divergence and per-quant MMLU ladder for Gemma 3 27B against an BF16 reference, both from the official Unsloth Dynamic 3.0 docs. KLD measures how far the quantized model’s token-distribution drifts from the unquantized one; MMLU 5-shot measures whether the model can still answer general-knowledge questions.
| Quant | File size | KL Divergence | MMLU 5-shot | Drop vs BF16 (71.5) |
|---|---|---|---|---|
| BF16 baseline | 51.4 GB | 0 | 71.50 | - |
| Q8_0 | 26.74 GB | (not listed) | 71.60 | +0.10 |
| Q6_K | 20.64 GB | (not listed) | 71.87 | +0.37 |
| Q5_K_M | 17.95 GB | (not listed) | 71.77 | +0.27 |
| Q4_K_XL | 15.64 GB | 0.0237 | 71.47 | -0.03 |
| Q4_K_M | 15.41 GB | 0.0249 | 71.23 | -0.27 |
| Q3_K_XL | 12.76 GB | 0.0806 | 70.87 | -0.63 |
| Q3_K_M | 12.51 GB | 0.0878 | 70.70 | -0.80 |
| IQ3_XXS | 10.07 GB | (not listed) | 68.27 | -3.23 |
| Q2_K | 9.78 GB | (not listed) | 68.50 | -3.00 |
| Q2_K_XL | 9.95 GB | 0.2209 | 68.70 | -2.80 |
| IQ2_M | 8.96 GB | 0.2582 | 66.47 | -5.03 |
| IQ2_XXS | 7.31 GB | 0.5210 | 59.20 | -12.30 |
| IQ1_M | 6.51 GB | 0.8000 | 48.10 | -23.40 |
| IQ1_S | 6.06 GB | 0.9729 | 41.87 | -29.63 |
Two things stand out. First, the cost is near-zero from BF16 down to Q4_K_M: about 0.3 percentage points on MMLU and a KLD of roughly 0.025. Unsloth frames small KLDs as “the closer the KL Divergence is to 0, the better (ie 0 means identical to the full precision model)” rather than as “near-imperceptible.” Second, the cost is not linear. Q4 to Q3 costs about 0.5 percentage points and triples the KLD. Q3 to Q2 costs another 2.4 percentage points and triples the KLD again. The “lossless until Q3, then sharp drop” pattern is the shape of the curve for most models.
The major quantization papers agree at the method level. GPTQ states “negligible accuracy degradation relative to the uncompressed baseline” at 3 to 4 bits. AWQ reports “protecting only 1% salient weights can greatly reduce quantization error”, the mechanism that lets 4-bit builds hold their accuracy. The Gemma 3 27B numbers above are consistent with both.
The damage is not uniform across tasks
MMLU is a knowledge benchmark. Real workloads do not look like MMLU. The same 0.27 percentage-point drop on MMLU that you barely notice in chat can show up as several percent of extra typos in code, missed tool calls in an agent loop, or invented function names in an editor - the failure modes that matter most are exactly the ones that degrade earliest.
Two patterns hold across every well-tested open model. First, code generation and exact recall degrade earlier than conversational fluency: a model can feel fine in chat and start breaking JSON output. Second, larger models tolerate quantization better than smaller ones at the same bit count. A 70B model at Q4 usually beats a 13B at Q8 at similar total size, which is why the VRAM tier guides push the parameter count up rather than the precision up.
Below roughly 4 bits per weight the damage is dominated by outliers. AWQ’s “1% salient weights” claim is the same observation stated mathematically: protect a few outlier channels at higher precision and the rest can be crushed. The IQ GGUF quants apply the same idea at the tensor level, which is why IQ4_XS holds up better than its nominal bit count predicts and why IQ2_XXS collapses.
The same number hides three different methods
A “4-bit” quantization is not a single number. AWQ, GPTQ, GGUF and MLX all claim 4-bit and reach it by different routes, so a “GGUF Q4_K_M”, a “GPTQ-Int4” and an “AWQ-Int4” cannot be compared by name. GPTQ uses a calibration dataset to compute a per-layer inverse-Hessian that distributes rounding error. AWQ scales activation outliers so the same bit width carries more signal. GGUF K-quants mix super-blocks with higher-precision scales for sensitive tensors. MLX uses Apple’s framework on Apple Silicon only. FP8 is 8-bit floating point, not 4-bit integer, and needs Hopper-generation NVIDIA or newer to run at hardware speed. It lives on its own cards rather than as a sibling of the GPTQ-Int4 release - for Qwen3.5-9B the GPTQ-Int4 repository ships only the three-part model-*.safetensors GPTQ-Int4 weights, while FP8 builds sit on separate community cards (e.g. RedHatAI’s Qwen3.5-9B-FP8-dynamic). The Gemma 3 27B ladder above is GGUF K-quants, KLD against BF16 and 5-shot MMLU: one model, one toolchain, one dataset.
How to measure the cost on your own workload
Two metrics matter, and perplexity is not one of them. Unsloth is explicit: “Using perplexity is incorrect since output token values can cancel out, so we must use KLD or harder benchmarks like Aider”. Perplexity sums log-probabilities across a corpus, so a model that over-weights one token and under-weights another can land on the same total and look fine while being wrong on every individual answer.
KL Divergence is the honest measure at the distribution level: how far the quantized model’s next-token probabilities are from the unquantized one, token by token. It needs both models loaded at once, the honest cost for a one-time calibration measurement.
For task-level measurement, three public benchmarks are reproducible:
- lm-evaluation-harness (EleutherAI) runs MMLU, GSM8K, HumanEval, ARC and dozens of other tasks. Run it against BF16 and your target quant; the per-task deltas are the answer.
- Aider Polyglot is the standard coding benchmark. DeepSeek V3.1’s Dynamic 3-bit GGUF scored 75.6 percent on it, one of the few published quant-vs-quals numbers for a frontier model.
- Divergence-300@32 is Unsloth’s newer metric: for the 32 most divergent tokens, measure accuracy drop.
Pick the model, pick three tasks you actually run, measure BF16 vs your target quant on those three. Within about 0.5 percentage points per task is noise. Outside that, you will notice.
What this changes about which quant to pick
The “Q4_K_M is the safe default” advice from the GGUF vs AWQ vs GPTQ vs MLX page survives contact with the numbers. From BF16 down to Q4_K_M on a 27B model, the MMLU delta is under 0.3 percentage points and the KLD is under 0.03. For chat, the loss is in the benchmark’s own noise floor.
Q5_K_M and Q6_K are not buying you much for chat. Their MMLU scores sit inside the noise band of Q4_K_M, and the llama.cpp quantize README shows they are noticeably slower at text generation than Q4. The reason to go above Q4 is a task that punishes rare-token errors: strict-typed code, structured extraction, agent tool-calling.
Q3 is the first level where the cost becomes visible. KLD triples versus Q4 and MMLU drops by about 0.7 percentage points. For a one-off chat you will not notice. For a benchmark sweep or a long agentic workflow you may. IQ4_XS is the smarter move when Q4_K_M does not quite fit - it stays above the 4-bit line and lands halfway between Q4 and Q3 on the cost axis.
Below Q3, you are paying real quality. Gemma 3 27B at IQ2_XXS loses 12 MMLU points versus BF16; at IQ1_S it loses 30. The only honest reason to use a sub-Q3 build is that you need that specific model for a task nothing smaller will do, and the only honest way to know that is to measure it on your task.
One procedural hazard from the llama.cpp quantize README: do not quantize an already-quantized file. The --allow-requantize flag exists and the docs warn it “can severely reduce quality compared to quantizing from 16bit or 32bit”. Two rounds of quantization compound the damage.
The bottom line
Quantization costs almost nothing from BF16 down to Q4_K_M, a small measurable amount from Q4 to Q3, and a lot below Q3. The Gemma 3 27B ladder is the cleanest published source for the shape of that curve, and the GPTQ and AWQ papers independently confirm the Q4-near-free finding at the method level. Cost is not uniform across tasks: chat hides damage, code and structured output expose it. Perplexity is the wrong metric; KL Divergence, lm-evaluation-harness and Aider Polyglot are the right ones. Measure on your workload before you trust a chart, and never re-quantize an already-quantized file.