Pick a model, and the next question is which of the twenty-five files in the repository to download. The names are not self-explanatory, the advice is contradictory, and the single most repeated rule of thumb is wrong.
This page covers what the formats actually are, what they really cost in disk and memory, and which one your runtime can load. If you have not chosen a model yet, start with which models fit your GPU.
The four-bit model is not four bits
The number in a quantization name is not the bits per weight. llama.cpp publishes the real figures, and they are consistently higher, because quantized formats store scaling factors and keep some tensors at higher precision.
| Build | Real bits/weight | Share of F16 |
|---|---|---|
| IQ4_XS | 4.4597 | 27.9% |
| Q4_K_S | 4.6672 | 29.2% |
| Q4_K_M | 4.8944 | 30.6% |
| Q5_K_M | 5.7036 | 35.6% |
| Q6_K | 6.5633 | 41.0% |
| Q8_0 | 8.5008 | 53.1% |
| F16 | 16.0005 | 100% |
So Q4_K_M is a 4.89-bit format and Q8_0 is an 8.5-bit format. The practical consequence is the rule of thumb everyone repeats, that Q4 is half the size of Q8, is wrong: the real saving is about 42%.
Real files agree. Here is the published ladder for Qwen3.5-9B, measured from the GGUF blobs rather than calculated:
| Build | Size | Share of BF16 |
|---|---|---|
| UD-IQ2_M | 3.65 GB | 20% |
| Q3_K_M | 4.67 GB | 26% |
| IQ4_XS | 5.17 GB | 29% |
| Q4_K_M | 5.68 GB | 32% |
| Q5_K_M | 6.58 GB | 37% |
| Q6_K | 7.46 GB | 42% |
| Q8_0 | 9.53 GB | 53% |
| BF16 | 17.92 GB | 100% |
Two independent sources, one a bits-per-weight measurement on Llama-3.1-8B and one a file listing for a different model from a different publisher, agree to within about a point at every step. That is the ratio to plan with.
The speed result that surprises people
Quantization is usually sold as a memory tradeoff you accept reluctantly. llama.cpp’s own benchmark table shows it is also a speed decision, and not in the direction most people assume.
| Build | Prompt processing (t/s) | Text generation (t/s) |
|---|---|---|
| Q2_K_S | 798.91 | 90.01 |
| Q4_K_M | 821.81 | 71.93 |
| Q6_K | 812.01 | 58.67 |
| Q8_0 | 865.09 | 50.93 |
| F16 | 923.49 | 29.17 |
Text generation gets slower as precision rises, from 90 tokens per second at Q2_K_S down to 29 at F16, a factor of three. Prompt processing does the opposite, and is fastest at F16.
The reason is that the two phases are bound by different things. Generating one token at a time reads the whole weight set per token, so it is limited by memory bandwidth, and a smaller file simply moves fewer bytes. Processing a prompt handles many tokens at once and is limited by arithmetic, where the dequantization step is extra work rather than a saving.
Two caveats. These are llama.cpp’s published figures for Llama-3.1-8B and the document does not state the hardware, so treat the ratios as the finding and not the absolute numbers. And this says nothing about output quality, which is the next section.
What it costs in quality, honestly
This is the part where most guides state a number and almost none cite one.
llama.cpp names perplexity and Kullback-Leibler divergence as the measures of quantization damage, but the comparison table it publishes reports size and speed, not either of those measures. So the widely repeated claims that Q4 costs “about 1% quality” or that Q5 is “nearly lossless” are not sourced from that table, and no primary source was found for them during this update.
What can be said without inventing figures:
- Damage rises as bits fall, and it rises faster below 4 bits per weight than above.
- It is not uniform across tasks. Code generation and exact recall degrade earlier than conversational fluency, which is why a model can feel fine in chat and start inventing function names in an editor.
- Larger models tolerate quantization better. A 70B model at Q4 usually beats a 13B at Q8 at similar total size.
- The
--allow-requantizepath is explicitly documented as harmful: llama.cpp warns it “can severely reduce quality compared to quantizing from 16bit or 32bit”. Quantize from full precision, never from an already quantized file.
If you need a defensible answer for your own use, measure it on your own task. Any single published number is either unsourced or measured on something that is not your workload.
The formats, and what can load them
The format question is mostly a runtime question. Pick the runtime first and the format follows.
| Format | Runtimes | Hardware | Typically published by |
|---|---|---|---|
| GGUF | llama.cpp, Ollama, LM Studio, Jan | CPU, CUDA, Metal, ROCm, Vulkan | Community, occasionally first-party |
| GPTQ | vLLM, SGLang, TGI | NVIDIA mainly | First-party for Qwen3.5 |
| AWQ | vLLM, SGLang, TGI | NVIDIA mainly | Community |
| FP8 | vLLM, SGLang | Hopper and newer NVIDIA | First-party for Qwen3.5 |
| MLX | mlx-lm, LM Studio | Apple Silicon only | Community |
GGUF is the default for single-user local inference and the only one of these that runs
well on CPU or splits a model between CPU and GPU. Its K quants are the mainstream
choice; IQ quants squeeze smaller at the same nominal bit count using an importance
matrix, at some speed cost.
GPTQ and AWQ are both 4-bit weight-only formats aimed at batched GPU serving, and in practice you choose by what your serving stack supports rather than by quality. Qwen publishes first-party GPTQ-Int4 builds for the Qwen3.5 line, including Qwen3.5-27B-GPTQ-Int4, while AWQ builds for the same models are community conversions.
FP8 is the one to know about if you have recent NVIDIA hardware, because it is roughly half the size of BF16 with hardware support rather than emulation, and Qwen publishes it first-party. It needs Hopper generation or newer.
MLX is Apple Silicon only and is the right answer there. It is not a competitor to GGUF elsewhere. Both engines now ship inside LM Studio and Ollama on Macs, so the choice is per model rather than per tool. Our Apple Silicon guide covers that case properly, because unified memory does not behave like VRAM.
What to actually download
On a consumer GPU, running one conversation at a time: Q4_K_M. It is the default for good reason, it is the best-supported build in every GGUF toolchain, and the speed table above shows it is also faster to generate with than Q6_K or Q8_0.
If Q4_K_M does not fit, go to IQ4_XS before Q3. On the Qwen3.5-9B ladder it is 5.17GB against Q4_K_M’s 5.68GB, which is often exactly the gap you need, and it stays above the four-bit line where damage starts accelerating.
Do not go below Q3 to make a bigger model fit. A smaller model at Q4 is usually the better trade. The tier guides on this site stop recommending a model at the point where the only way to fit it is a sub-Q4 build, and that is the reason.
Q6_K and Q8_0 are for when you have memory spare and a task that punishes errors, which in practice means code generation and structured extraction. You are paying real generation speed for it.
Serving several users at once: GPTQ-Int4, AWQ or FP8 under vLLM, not GGUF. Batched serving is a different problem from the one llama.cpp is optimised for.
The last thing worth internalising is that quantization names are not comparable across formats. A GGUF Q4_K_M, a GPTQ-Int4 and an MLX 4-bit are three different methods that share a marketing number, and none of them stores four bits per weight.