AI hardware · multi-GPU

Will the model fit across several GPUs?

List up to four cards, pick a model and context; see pooled memory, the fit check and a suggested split.

Inputs

Starting values are illustrative, not current prices or specifications. Replace each with your own.

GPU 1 memory (GiB)

GPU 2 memory (GiB)

GPU 3 memory (GiB, optional)

GPU 4 memory (GiB, optional)

Reserve per GPU (GiB)Assumption, not a published figure: runtime and display use on each card.

Model presetFills the four fields below from each model's Hugging Face config.json. Overwrite any of them for another model.

Parameters (billions)For a mixture-of-experts model enter the total parameters, because every expert has to be in memory (active parameters apply only to the tokens-per-second estimate).

Layers

KV headsnum_key_value_heads in config.json (fewer than the attention heads for grouped-query models).

Head dimensionhead_dim, or hidden_size divided by num_attention_heads.

Weight format

Custom bits per weight (optional)Overrides the format above, for example the bits per weight of a specific file: file size x 8 / parameters.

Context length (tokens)

Parallel sequences

KV cache type

Result

Answer

With llama.cpp's default layer split, the layers and the KV cache are divided across the GPUs, so the pooled memory is the sum of each card's usable memory. The model fits if weights plus KV cache are no more than that sum. This is a fit check only; it does not predict speed.

Worked example

Llama 3.3 70B (70.554 billion parameters) at Q4_K is 36.96 GiB of weights, and the 8,192-token f16 cache is 2.50 GiB, so 39.46 GiB is needed. Two 24 GiB cards with 1 GiB reserved each give 46.00 GiB usable, leaving 6.54 GiB spare, so it fits on paper. A single 24 GiB card has 23.00 GiB usable and does not. The suggested split is 23,23.

Formulas and notes

Usable per card = card memory − reserve (not below zero). Pooled usable memory is the sum.
Needed = weights + KV cache (formulas as in the VRAM and KV cache calculators).
Per-card share is proportional to usable memory, and the suggested split is those usable amounts as comma-separated proportions. llama.cpp's server documents --tensor-split as the "fraction of the model to offload to each GPU, comma-separated list of proportions, e.g. 3,1", and --split-mode layer (default) as splitting layers and KV across GPUs.
A card whose usable memory is smaller than one layer (weights ÷ layers) is flagged, since layers are the unit of the split.
Splitting adds some extra buffers and cross-card transfer that this estimate does not model. Tools other than llama.cpp may split differently.

Planning estimate only, not a benchmark: real memory use and speed depend on the model file, the runtime, the driver and the hardware. Sources, opened 2026-10-08: llama.cpp, server README (cache types, GPU split options) · Databricks, LLM Inference Performance Engineering: Best Practices (12 Oct 2023) · Hugging Face Hub documentation, GGUF · llama.cpp source, ggml-common.h (block_q8_0 and block_q4_0 structs) · Hugging Face, unsloth/Llama-3.3-70B-Instruct (config.json, parameter count)

FAQ

Can I pool VRAM from two GPUs to run a bigger model?

With llama.cpp's layer split, yes: the layers and the KV cache are spread across the cards, so the total usable memory is what counts. Other software may differ, and the cards need not be identical.

Does a second GPU double my tokens per second?

Do not assume so. The llama.cpp server README labels the default layer split "pipelined" and the row split "parallelized". A layer split mainly adds capacity; measure before expecting a speedup.

What is the tensor split value for?

llama.cpp takes --tensor-split as proportions of the model per GPU. The suggestion is proportional to each card's usable memory, a starting point you can adjust.

Why reserve memory on every card?

Some of each card's memory may be taken by the runtime or a display. The 1 GiB default is an assumption you should replace with what your runtime reports.

What if the model fits only barely?

Leave margin. This tool ignores extra buffers and transfer overhead, so a headroom close to zero is likely to fail in practice. Try a smaller quantization or a shorter context.