AI hardware · quantization
Compare quantization file sizes.
Enter a parameter count; get the approximate weight size for FP16, Q8_0, Q6_K, Q5_K, Q4_K, IQ4_XS, Q3_K and Q2_K.
Inputs
Starting values are illustrative, not current prices or specifications. Replace each with your own.
Result
Answer
A quantized file is about parameters × bits per weight ÷ 8 bytes. FP16 uses 16 bits, Q8_0 about 8.5, Q6_K 6.5625, Q5_K 5.5 and Q4_K 4.5, so moving from FP16 to Q4_K cuts the size to about 28 percent. Real files named Q4_K_M or Q5_K_M mix tensor types and come out somewhat larger than the base figure.
Worked example
Llama 3.1 8B (8.03 billion parameters): FP16 is 16.06 GB (14.96 GiB), Q8_0 is 8.53 GB, Q6_K 6.59 GB, Q5_K 5.52 GB and Q4_K 4.52 GB (4.21 GiB), which is 28 percent of FP16.
Formulas and notes
- Size in bytes = parameters × bits per weight ÷ 8.
- Bits per weight, Hugging Face GGUF documentation: Q6_K 6.5625, Q5_K 5.5, Q4_K 4.5, Q3_K 3.4375, Q2_K 2.625, IQ4_XS 4.25. FP16 is 16 bits. Q8_0 is 8.5 because the ggml block holds one 16-bit scale and 32 8-bit values (34 bytes per 32 weights).
- The quantize tool's
--pureoption "quantizes all tensors to the same type" and disables k-quant mixtures, so the usual _M and _S files are mixtures and differ from these base figures. - The quantize tool can set the embedding and output tensors to different types (--token-embedding-type, --output-tensor-type), and files carry metadata, so real sizes differ a little. Compare with the real file size on the model page before downloading.
- This tool says nothing about output quality; lower bit counts lose more accuracy, and the amount depends on the model.
Planning estimate only, not a benchmark: real memory use and speed depend on the model file, the runtime, the driver and the hardware. Sources, opened 2026-10-08: Hugging Face Hub documentation, GGUF · llama.cpp source, ggml-common.h (block_q8_0 and block_q4_0 structs) · llama.cpp, quantize tool README · NIST, Prefixes for binary multiples · Hugging Face, unsloth/Meta-Llama-3.1-8B (config.json, parameter count)
FAQ
How big is a Q4 model compared with FP16?
About 28 percent of the size, from 4.5 bits per weight against 16. For 8.03 billion parameters that is 4.52 GB against 16.06 GB.
Why is my Q4_K_M file bigger than this table?
The _M and _S names are mixtures of quant types across tensors, not a single type. The quantize tool's --pure option exists to turn the mixtures off, which shows they are the default. Use the exact file size from the model page.
Why is Q8_0 8.5 bits and not 8?
Each Q8_0 block stores 32 8-bit values plus one 16-bit scale, which is 34 bytes per 32 weights, or 8.5 bits per weight (ggml block_q8_0).
What is the difference between GB and GiB here?
A GB is 10^9 bytes and a GiB is 2^30 bytes, 1,073,741,824 bytes (NIST). Both are shown because download pages usually use one and GPU tools the other.
Which quantization should I pick?
This tool only reports size. A common approach is the largest format whose weights plus cache fit your memory, then testing the output on your own prompts.