AI hardware · KV cache
How big is the KV cache?
Layers, KV heads, head dimension and context length in; cache size per token and in total out, for f16, q8_0 and q4_0 caches.
Inputs
Starting values are illustrative, not current prices or specifications. Replace each with your own.
Result
Answer
KV cache size is 2 (one for keys, one for values) × layers × KV heads × head dimension × tokens × bytes per element. Models that use grouped-query attention share keys and values across attention heads, so the KV heads count in config.json is smaller than the attention heads count and the cache is smaller (Databricks).
Worked example
Llama 3.1 8B has 32 layers, 8 KV heads and a head dimension of 128. Per token in f16 that is 2 × 32 × 8 × 128 × 2 bytes = 131,072 bytes, or 128 KiB. For 8,192 tokens that is 1.00 GiB; for the model's 131,072-token limit it is 16.00 GiB. A q8_0 cache at 8,192 tokens is 0.53 GiB and q4_0 is 0.28 GiB.
Formulas and notes
- Cache per token = 2 × layers × KV heads × head dimension × bytes per element. Databricks writes the same thing as batch size × sequence length × (d_model ÷ n_heads) × layers × 2 × bytes × n_kv_heads.
- NVIDIA's simpler form, 2 × layers × (heads × head dimension) × precision, is the special case where every head has its own keys and values.
- Bytes per element: f32 = 4, f16 = 2. For q8_0 and q4_0 the ggml block layout stores one 16-bit scale per 32 values, giving 34/32 = 1.0625 and 18/32 = 0.5625 bytes. llama.cpp lists f16 as the default and q8_0 and q4_0 among the allowed cache types.
- A quantized cache stores fewer bits per element; this tool does not estimate any effect on output quality.
Planning estimate only, not a benchmark: real memory use and speed depend on the model file, the runtime, the driver and the hardware. Sources, opened 2026-10-08: Databricks, LLM Inference Performance Engineering: Best Practices (12 Oct 2023) · NVIDIA Technical Blog, Mastering LLM Techniques: Inference Optimization (17 Nov 2023) · llama.cpp source, ggml-common.h (block_q8_0 and block_q4_0 structs) · llama.cpp, server README (cache types, GPU split options) · Hugging Face, unsloth/Meta-Llama-3.1-8B (config.json, parameter count) · Hugging Face, Qwen/Qwen2.5-7B (config.json, parameter count) · Hugging Face, Qwen/Qwen2.5-32B (config.json, parameter count) · Hugging Face, unsloth/Llama-3.3-70B-Instruct (config.json, parameter count)
FAQ
How do I calculate KV cache size?
Multiply 2 (keys and values) by the number of layers, the number of KV heads, the head dimension, the number of tokens and the bytes per element. Layers, KV heads and head dimension are in the model's config.json.
Why is the KV heads number smaller than the attention heads?
Grouped-query attention shares keys and values among groups of query heads, which keeps the cache smaller (Databricks). When there is one KV head it is multi-query attention.
How big is the KV cache for a 70B model?
For Llama 3.3 70B (80 layers, 8 KV heads, head dimension 128) at 8,192 tokens in f16 this page gives 2.50 GiB.
Does the KV cache scale with the number of users?
Yes. NVIDIA notes that with batching each request's cache is allocated separately, so total cache grows with batch size as well as sequence length. Use the parallel sequences field.
Can I shrink the KV cache?
Cache types such as q8_0 and q4_0 store fewer bits per element, as the rows in the result show. Whether the quality cost is acceptable depends on your model and task, which this calculator does not test.