A/B test significance calculator.

How the test works

The null hypothesis is that both variants share one true conversion rate. Pool the data: p̂ = (x_A + x_B) / (n_A + n_B). The test statistic is

z = (p̂_B − p̂_A) / √( p̂(1 − p̂)(1/n_A + 1/n_B) )

For a two-sided test the p-value is 2 × P(Z > |z|); for a one-sided test where the alternative is “B is better” it is P(Z > z). The result is called significant when the p-value is below α = 1 − confidence level. The confidence interval for the difference uses the unpooled standard error: (p̂_B − p̂_A) ± z_1−α/2 √( p̂_A(1−p̂_A)/n_A + p̂_B(1−p̂_B)/n_B ). Relative lift is the difference divided by A's rate.

Worked example

A: 200 of 1,000 = 20.0%. B: 240 of 1,000 = 24.0%. Pooled p̂ = 440 / 2,000 = 0.22. Standard error = √(0.22 × 0.78 × 0.002) = 0.018526. z = 0.04 / 0.018526 = 2.159. Two-sided p = 2 × P(Z > 2.159) = 0.0308, below 0.05, so significant at 95% but not at 99% (p is above 0.01). The unpooled standard error is 0.018504, so the 95% interval is 0.04 ± 1.960 × 0.018504 = +0.0037 to +0.0763, or +0.37 to +7.63 percentage points. Relative lift = 0.04 / 0.20 = 20.0%.

Limits

The normal approximation needs reasonably large samples. NIST gives min{Np₀, N(1 − p₀)} ≥ 5 as a criterion for its one-sample proportion test; this page applies the same check to the pooled expected successes and failures in each variant and warns when it fails. That application is a common rule of thumb, not a published A/B-testing standard. For small samples NIST calls the Fisher exact test an excellent choice. Decide the sample size in advance (see the sample size calculator) and read the p-value once the planned traffic is in: checking repeatedly and stopping at the first significant reading raises the false-positive rate. Statistical significance says the difference is unlikely to be chance under the model, not that it is large or valuable; the confidence interval shows how big it might be.

Sources, opened 8 October 2026: NIST/SEMATECH e-Handbook of Statistical Methods, 7.3.3 How can we determine whether two processes produce the same proportion of defectives? (pooled z statistic, one- and two-sided comparison, Fisher exact test for small samples); Wikipedia, Z-test (two-proportion Z-test and the confidence interval for the difference of two proportions); NIST/SEMATECH e-Handbook, 7.1.3.1 Critical values and p values (definition of the p-value and comparing it with α); NIST/SEMATECH e-Handbook, 7.2.4 Does the proportion of defectives meet requirements? (the min{Np₀, N(1−p₀)} ≥ 5 criterion).