Utilization vs latency calculator.

Worked example

A single worker takes 100 ms per request. At 50% utilization the mean response time is 100 / (1 - 0.5) = 200 ms. At 90% it is 1,000 ms, ten times the service time. To keep the mean at or under 200 ms the worker can run at most 50.0% busy, about 5.0 requests per second. At the same utilization more workers help: they share one queue, so fewer requests wait.

How it works

The tool uses the M/M/c mean response time W = C(c, a) / (c x mu - lambda) + 1 / mu. For c = 1 this reduces to W = 1 / (mu - lambda) = service time / (1 - rho). For each utilization rho it sets lambda = rho x c x mu. The highest utilization for a target is found by bisection because W only increases with load.

Reading it. The mean is not the 95th or 99th percentile; tail latency at the same load is worse. Treat the curve as the reason to leave headroom, not as a precise forecast. Requests that are not exponential in length or arrive in bursts reach the knee sooner.

FAQ

Why does latency rise so fast at high utilization?

Waiting time depends on spare capacity. A server at 90% busy has 10% spare, at 99% only 1%, so queues build much faster than the load itself grows.

What utilization should I target?

It depends on the latency target and how bursty traffic is. This tool shows the utilization where the mean meets your target; set a lower number for percentile targets and bursts.

Does adding servers fix high utilization latency?

It moves the knee: with more servers sharing one queue, the same utilization gives a shorter mean wait. The curve still rises steeply as you approach 100%.

Is this the mean or the percentile?

The mean response time. Tail percentiles are higher than the mean, so a percentile target needs a lower utilization than the one shown for the same number.