AI hardware · VRAM

How much VRAM does your local LLM need?

Parameters, weight format and context length in; weights, KV cache and total GPU memory out, with a check against your card.

Inputs

Starting values are illustrative, not current prices or specifications. Replace each with your own.

Model presetFills the four fields below from each model's Hugging Face config.json. Overwrite any of them for another model.

Parameters (billions)For a mixture-of-experts model enter the total parameters, because every expert has to be in memory (active parameters apply only to the tokens-per-second estimate).

Layers

KV headsnum_key_value_heads in config.json (fewer than the attention heads for grouped-query models).

Head dimensionhead_dim, or hidden_size divided by num_attention_heads.

Weight format

Custom bits per weight (optional)Overrides the format above, for example the bits per weight of a specific file: file size x 8 / parameters.

Context length (tokens)

Parallel sequences

KV cache type

Runtime overhead (GiB)Assumption, not a published figure: space for compute buffers and the GPU runtime. Raise it if your runtime reports more.

Your GPU memory (GiB, optional)Enter the listed size of the card. This tool treats 12 GB as 12 GiB.

Result

Answer

Memory for a local LLM is the model weights plus the KV cache plus runtime overhead. The weights are parameters multiplied by bits per weight divided by 8; NVIDIA's inference guide gives the 16-bit case as a 7 billion parameter model taking roughly 7B x 2 bytes, about 14 GB. The KV cache grows with context length, so a longer context needs more memory on top of the same weights.

Worked example

Llama 3.1 8B (8.03 billion parameters) at Q4_K, 4.5 bits per weight: 8.03 × 10⁹ × 4.5 ÷ 8 = 4.517 GB, which is 4.21 GiB. With an 8,192-token context the f16 KV cache is 1.00 GiB (see the KV cache calculator). Adding the assumed 1 GiB overhead gives 6.21 GiB in total, so a 12 GiB card has 5.79 GiB to spare.

Formulas and notes

Weights = parameters × bits per weight ÷ 8.
KV cache = 2 × layers × KV heads × head dimension × tokens × bytes per element (Databricks).
Total = weights + KV cache + overhead. The overhead is your assumption.
Bits per weight for K-quants and IQ4_XS come from the Hugging Face GGUF documentation; Q8_0 is 8.5 from the ggml block layout (one 16-bit scale per 32 8-bit values). Files named Q4_K_M or Q5_K_M are k-quant mixtures (the quantize tool has a --pure option to turn mixtures off), so they run somewhat above the base figure; use the custom field if you know the real file size.
Sizes are shown in GiB (2³⁰ bytes) and GB (10⁹ bytes), the distinction NIST defines.

Planning estimate only, not a benchmark: real memory use and speed depend on the model file, the runtime, the driver and the hardware. Sources, opened 2026-10-08: NVIDIA Technical Blog, Mastering LLM Techniques: Inference Optimization (17 Nov 2023) · Databricks, LLM Inference Performance Engineering: Best Practices (12 Oct 2023) · Hugging Face Hub documentation, GGUF · llama.cpp source, ggml-common.h (block_q8_0 and block_q4_0 structs) · llama.cpp, quantize tool README · NIST, Prefixes for binary multiples · Hugging Face, unsloth/Meta-Llama-3.1-8B (config.json, parameter count) · Hugging Face, Qwen/Qwen2.5-7B (config.json, parameter count) · Hugging Face, Qwen/Qwen2.5-32B (config.json, parameter count) · Hugging Face, unsloth/Llama-3.3-70B-Instruct (config.json, parameter count)

FAQ

How much VRAM do I need to run a 7B or 8B model?

For weights alone, multiply parameters by bytes per weight. NVIDIA's guide puts a 7B model in 16-bit at about 14 GB. For an 8B model at Q4_K (4.5 bits per weight) this page computes 4.52 GB of weights, and 16.06 GB at FP16. Add the KV cache for your context and some overhead.

How much VRAM does a 70B model need?

At Q4_K the weights of a 70.554 billion parameter model are 39.7 GB (37.0 GiB) before any KV cache, which is more than a single 24 GiB card holds. See the multi-GPU VRAM calculator for splitting it across cards.

Does a longer context use more VRAM?

Yes. The KV cache grows linearly with sequence length and with the number of sequences (NVIDIA), while the weights stay the same. Use the KV cache calculator to see the cache on its own.

Where does the 1 GiB runtime overhead come from?

It is an assumption, not a published figure. It stands in for everything besides the weights and the KV cache, which depends on your software and settings. Replace it with the number your runtime prints when it loads the model.

Are these numbers a benchmark?

No. They are arithmetic from the formulas above. Check your runtime's own load log or nvidia-smi for the real figure.