AI hardware · decode speed
How fast can your hardware generate tokens?
Memory bandwidth and model size in; the ceiling on single-stream decode speed, with what you get at 70 and 50 percent of it.
Inputs
Starting values are illustrative, not current prices or specifications. Replace each with your own.
Result
Answer
At small batch sizes, generating each token means reading all the model weights (plus the KV cache) from memory, so decode speed is limited by memory bandwidth, not compute. The ceiling is bandwidth divided by bytes read per token. Real systems reach only a share of it, so treat the result as an upper bound, not a prediction.
Worked example
An 8.03 billion parameter model at Q4_K is 4.517 GB. On hardware with 1,000 GB/s of memory bandwidth the ceiling is 1,000 ÷ 4.517 = 221.4 tokens per second; at 70 percent of bandwidth 155.0, at 50 percent 110.7. The 70.554 billion parameter model at Q4_K reads 39.7 GB per token, a ceiling of 25.2 tokens per second on the same bandwidth.
Formulas and notes
- Tokens per second (ceiling) = memory bandwidth ÷ (model bytes + KV cache bytes read per token).
- Databricks describes the same relation as Model Bandwidth Utilization: achieved bandwidth is (model size + KV cache size) ÷ time per output token, and it gives a 7B model at 16-bit with 14 ms per token as 14 GB in 14 ms, 1 TB/s. Databricks says decode at small batch sizes is limited by how fast parameters load from memory, and NVIDIA calls the decode phase memory-bound.
- For a mixture-of-experts model enter the active parameters used per token, because only those weights are read; this is a consequence of the formula, not a published rule for any one model.
- Published bandwidth examples: Apple lists 800GB/s for the M2 Ultra. Check your own spec sheet; GPU names alone are not enough to know a figure.
- This is not a benchmark. Databricks says generating the first token is typically compute-bound, so prompt processing is not covered, and batching changes the picture.
Planning estimate only, not a benchmark: real memory use and speed depend on the model file, the runtime, the driver and the hardware. Sources, opened 2026-10-08: Databricks, LLM Inference Performance Engineering: Best Practices (12 Oct 2023) · NVIDIA Technical Blog, Mastering LLM Techniques: Inference Optimization (17 Nov 2023) · Apple Newsroom, Apple introduces M2 Ultra (June 2023) · Hugging Face Hub documentation, GGUF · Hugging Face, unsloth/Meta-Llama-3.1-8B (config.json, parameter count)
FAQ
How do I estimate tokens per second for a local LLM?
Divide your memory bandwidth in GB/s by the model size in GB (plus the KV cache read per token). The result is the most tokens per second a single stream can reach.
Why is memory bandwidth the limit and not compute?
Databricks explains that at small batch sizes the speed depends on how fast model parameters load from memory to the compute units, not on how fast the arithmetic runs. NVIDIA calls the decode phase memory-bound for the same reason.
Why is my real speed lower than the result?
The formula is a ceiling. Databricks defines Model Bandwidth Utilization because achieved bandwidth is below the peak (its example runs at 50 percent). The 70 and 50 percent rows show what lower utilization means.
Does a smaller quantization run faster?
Under this model, yes: fewer bytes per token means a higher ceiling. In the example, Q4_K reads about 28 percent of the bytes FP16 does.
Does this apply to mixture-of-experts models?
Enter the active parameters per token rather than the total, since only the active experts are read for each token. Check the model card for the active count.