AI hardware · context

How much context fits in your VRAM?

GPU memory, model and weight format in; the number of KV cache tokens that fit after the weights and a reserve.

Inputs

Starting values are illustrative, not current prices or specifications. Replace each with your own.

GPU memory (GiB)Enter the listed size of the card. This tool treats 24 GB as 24 GiB.

Reserve for runtime and desktop (GiB)Assumption, not a published figure. Raise it if the card also drives a display.

Model presetFills the four fields below from each model's Hugging Face config.json. Overwrite any of them for another model.

Parameters (billions)For a mixture-of-experts model enter the total parameters, because every expert has to be in memory (active parameters apply only to the tokens-per-second estimate).

Layers

KV headsnum_key_value_heads in config.json (fewer than the attention heads for grouped-query models).

Head dimensionhead_dim, or hidden_size divided by num_attention_heads.

Weight format

Custom bits per weight (optional)Overrides the format above, for example the bits per weight of a specific file: file size x 8 / parameters.

Parallel sequences

KV cache type

Model context limit (tokens, optional)max_position_embeddings in config.json. Memory may allow more tokens than the model supports.

Result

Answer

The context that fits is the memory left after the weights and a reserve, divided by the KV cache bytes per token. Memory is only one limit: the model also has its own maximum context, listed in its config.json.

Worked example

Qwen2.5 32B (32.764 billion parameters, 64 layers, 8 KV heads, head dimension 128) at Q4_K weighs 17.16 GiB. On a 24 GiB card with a 1 GiB reserve that leaves 5.84 GiB. The f16 cache costs 2 × 64 × 8 × 128 × 2 = 262,144 bytes (256 KiB) per token, so 23,904 tokens fit. With a q8_0 cache the same card holds 44,995 tokens.

Formulas and notes

Room for cache = GPU memory − reserve − weights.
Tokens = room ÷ (KV bytes per token × parallel sequences), rounded down.
KV bytes per token = 2 × layers × KV heads × head dimension × bytes per element (Databricks).
If the weights alone exceed the budget the tool says so rather than showing a negative context.
Some runtimes size other buffers by context length too, so treat the result as an optimistic ceiling.

Planning estimate only, not a benchmark: real memory use and speed depend on the model file, the runtime, the driver and the hardware. Sources, opened 2026-10-08: Databricks, LLM Inference Performance Engineering: Best Practices (12 Oct 2023) · Hugging Face Hub documentation, GGUF · llama.cpp source, ggml-common.h (block_q8_0 and block_q4_0 structs) · llama.cpp, server README (cache types, GPU split options) · NIST, Prefixes for binary multiples · Hugging Face, unsloth/Meta-Llama-3.1-8B (config.json, parameter count) · Hugging Face, Qwen/Qwen2.5-7B (config.json, parameter count) · Hugging Face, Qwen/Qwen2.5-32B (config.json, parameter count) · Hugging Face, unsloth/Llama-3.3-70B-Instruct (config.json, parameter count)

FAQ

How do I find the maximum context length for my GPU?

Subtract the weights and a reserve from your GPU memory, then divide what is left by the KV cache bytes per token. That is the calculation this page does.

Why does my runtime report a smaller context than this tool?

This tool lumps everything besides the weights and the cache into the one reserve you enter, and a runtime may use more. Increase the reserve to match what your runtime prints at load.

Does the model have its own context limit?

Yes. The model's config.json lists max_position_embeddings. Enter it in the optional field and the tool caps the answer when memory would allow more.

What does a q8_0 or q4_0 cache change?

It stores fewer bits per cached element, so more tokens fit in the same memory. In the example, q8_0 raises the fit from 23,904 to 44,995 tokens. The quality effect is not estimated here.

What if I serve several users at once?

Enter the number of parallel sequences. Each sequence needs its own cache, so the tokens per sequence fall as the count rises.