This site’s guide to which quantization format to pick ended on an admission: the claims that Q4 costs “about 1% quality” or that Q5 is “nearly lossless” had no primary source behind them. What actually degrades, and how anyone would notice, is what it left open. Measured answers exist, and they travel badly.
One rule governs the rest. Perplexity and KL-divergence measure how far a quantized model’s output distribution has moved from full precision; accuracy measures how often answers are right. Neither becomes the other here, and every figure names its model, method and bit width, benchmark and publisher.
8-Bit Is Close to Lossless, With One Exception on Record
llama.cpp’s own LLaMA 3 8b Scoreboard puts GGUF q8_0 of LLaMA-3-8B at WikiText-2 perplexity 6.234284 against an F16 baseline of 6.233160, KL-divergence 0.001355. Project-published distribution figures, not accuracy.
For accuracy, Mekala et al. (arXiv:2505.20276) covered Llama-3.1 8B and 70B and Qwen-2.5 7B, 32B and 72B on five long-context benchmarks. The paper’s own cross-model, cross-task average puts FP8 at a 0.2% drop and GPTQ-int8 at 0.8% against BF16. The exception is specific: in the same study, Llama-3.1-70B under GPTQ-int8 drops 11.1% on NoCha, a book-length claim-verification benchmark, with no comparable decline on Ruler or OneRuler under the identical method.
The 4-Bit Fork: Same Method, Two Similar Models, Opposite Signs
The most useful result in the literature is a contrast. On OneRuler, a multilingual needle-in-a-haystack benchmark, Mekala et al. report that “BNB-nf4 quantization leads to a 59% accuracy drop for Llama-3.1 70B, whereas the similarly sized Qwen-2.5 72B gains 0.6% with the same method.” Same method, same benchmark, same nominal size, different families: one collapses, one improves slightly.
The paper is not internally consistent about the size of that collapse. Its main text gives 59%, an average over the 64K and 128K context lengths. Appendix C.2 says BNB-nf4 “performs particularly poorly on Llama-3.1 70B, with a 66% drop in accuracy” on the same benchmark. Both are verbatim from one paper with no stated aggregation difference, so the citable figure is its own range of 59 to 66 percent.
On Ruler, the English retrieval benchmark in that study, GPTQ-int4 costs Llama-3.1-8B 21.2%, and BNB-nf4 costs Llama-3.1-8B 30.7%, Qwen-2.5-7B 29.7% and Qwen-2.5-32B 15.7%.
Method matters as much as bit count. Huang et al. (arXiv:2404.14047) measured LLaMA3-8B at 4-bit, group size 128, on a five-task CommonSenseQA average: data-free RTN scores 63.9, GPTQ 67.3, AWQ 68.2, against an FP16 baseline of 68.6. Their perplexity numbers stay separate: WikiText2 6.1 at FP16, 8.5 RTN, 6.5 GPTQ, 6.6 AWQ.
Below 4 Bits, Generic Methods Fall Over
Huang et al. put it plainly: “under 2-3 bits, GPTQ causes severe accuracy collapse when quantized LLaMA3,” while the methods built for low-bit regimes, in their naming AWQ, QuIP and SliM-LLM, “can keep LLaMA3’s capability at 3-bit and even push the 2-bit quantization to promising.” Whether the method was built for that regime decides the outcome, not the bit count. The llama.cpp scoreboard shows the same in distribution terms: GGUF q2_K of LLaMA-3-8B, quantized without an importance matrix, records WikiText-2 perplexity 9.751568 and KL-divergence 0.445132 against an F16 baseline of 6.233160. Those are distribution numbers, not an accuracy claim.
Small Models Pay a Larger Relative Cost
Huang et al. ran the same methods at 4-bit, group size 128, on LLaMA3-70B: RTN 74.3 against an FP16 baseline of 75.3. Subtracting their own published numbers, RTN costs the 8B model 4.7 points on that benchmark and the 70B model 1.0 point.
Meta’s model card for its quantized Llama 3.2 releases, the vendor’s own numbers on its own methods, shows the same at the small end. On IFEval, Llama-3.2-1B-Instruct under a Vanilla PTQ configuration Meta measured but did not release scores 51.5 against a bf16 baseline of 59.5, and 14.3 on BFCL V2 tool use against 25.7. The 3B under that configuration scores 73.9 on IFEval against 77.4.
Long-Context Recall Is the Least Tested Failure Mode
The sharpest illustration is an absence. Meta’s card marks every long-context benchmark N/A for every quantized Llama 3.2 configuration: InfiniteBench En.QA is 20.3 for the 1B at bf16 and N/A for Vanilla PTQ, SpinQuant and QLoRA alike. The one vendor shipping official quantized weights did not evaluate long context on them, and cut the supported window from 128K to 8K outright. That was a product decision, not a measurement, per the Meta AI announcement: “given the limited runtime memory available on mobile devices, we prioritized short-context applications up to 8K for these new quantized models.”
Mekala et al.’s conclusion is the counterweight: “while prior work often suggests 8-bit quantization is nearly lossless and 4-bit methods are good enough, our findings demonstrate that the impact of quantization becomes more pronounced in long-context scenarios.” Their proposed cause, hedged by the authors as “a possible explanation,” is accumulated error in RoPE position embeddings.
The Measured Figures
| Model | Method (bit width) | Benchmark | Result vs baseline | Published by |
|---|---|---|---|---|
| LLaMA3-8B | RTN, 4-bit g128 | CommonSenseQA 5-task avg | 63.9 vs 68.6 FP16 | Huang et al. |
| LLaMA3-8B | GPTQ, 4-bit g128 | CommonSenseQA 5-task avg | 67.3 vs 68.6 FP16 | Huang et al. |
| LLaMA3-8B | AWQ, 4-bit g128 | CommonSenseQA 5-task avg | 68.2 vs 68.6 FP16 | Huang et al. |
| LLaMA3-70B | RTN, 4-bit g128 | CommonSenseQA 5-task avg | 74.3 vs 75.3 FP16 | Huang et al. |
| Llama-3.1-70B | GPTQ-int8 (W8A16) | NoCha | 11.1% drop vs BF16 | Mekala et al. |
| Llama-3.1-70B | BNB-nf4 (W4A16) | OneRuler, 64K+128K | 59% drop vs BF16 (appendix: 66%) | Mekala et al. |
| Qwen-2.5-72B | BNB-nf4 (W4A16) | OneRuler | 0.6% gain vs BF16 | Mekala et al. |
| Llama-3.1-8B | GPTQ-int4 (W4A16) | Ruler, 64K+128K | 21.2% drop vs BF16 | Mekala et al. |
| Llama-3.1-8B | BNB-nf4 (W4A16) | Ruler, 64K+128K | 30.7% drop vs BF16 | Mekala et al. |
| Llama-3.2-1B-Instruct | Vanilla PTQ, W4 g32 A8 | IFEval | 51.5 vs 59.5 bf16 | Meta, own card |
| Llama-3.2-1B-Instruct | Vanilla PTQ, W4 g32 A8 | BFCL V2 (tool use) | 14.3 vs 25.7 bf16 | Meta, own card |
| Llama-3.2-3B-Instruct | Vanilla PTQ, W4 g32 A8 | IFEval | 73.9 vs 77.4 bf16 | Meta, own card |
| LLaMA-3-8B | GGUF q2_K, no imatrix | WikiText-2 PPL, KLD | 9.751568 and 0.445132 (F16: 6.233160) | llama.cpp |
| LLaMA-3-8B | GGUF q8_0 | WikiText-2 PPL, KLD | 6.234284 and 0.001355 (F16: 6.233160) | llama.cpp |
The last two rows are distribution measurements; the rest are accuracy. The two are never mixed.
How to Test Your Own Setup
No published number substitutes for a run on the actual model, method and task. Four steps make that cheap.
Dump a baseline. Per its own README, llama.cpp’s perplexity tool runs the F16 build once with --kl-divergence-base pointing at an output file, then each quantized build of the same model against that file with --kl-divergence. Swap the conventional Wikitext-2 text for text from the real domain.
Build a held-out prompt set from the task itself: tool calls, code completions, exact quotations, non-English prompts where relevant.
Test at the real context length. A 2K spot check will not surface a retrieval failure at 64K. Bury a known answer partway through a document of the length actually used and check whether it comes back.
Compare against a higher-precision run of the same model, same prompts, temperature 0, since every source here computes quantized minus baseline. Quantize down from the original weights: llama.cpp’s quantize documentation warns that requantizing an already-quantized file “can severely reduce quality compared to quantizing from 16bit or 32bit.” That applies to the test builds too.
What Is Not Covered Here
No arithmetic-specific figure appears above, because none was found broken out by method and bit width. Nothing is claimed about refusal or safety behaviour, which none of these sources measured. No MLX figure appears either: nothing found quantifies Apple Silicon quantization at matched bit widths.
What This Means
The shape holds across every source above. 8-bit measures close to lossless on distribution metrics, on short-form accuracy and on long-context tasks, with the Llama-3.1-70B NoCha result as the documented exception. 4-bit is where the decision sits, and its cost is not a percentage that travels: one method on two models of the same nominal size produced a 59 to 66 percent drop on one and a 0.6% gain on the other. Below 4 bits, purpose-built methods hold and generic ones collapse, smaller models lose proportionally more, and long-context recall breaks first while getting benchmarked last.
The Bottom Line
There is no defensible “Q4 costs about 1%” number, and not for want of measurement. Everyone who measured across more than one model or task found a spread wide enough to make an average meaningless. 8-bit is close to free. 4-bit depends on the model, the method and the task, and the only way to know is a held-out prompt set at the real context length against a higher-precision run of the same weights.