A/B Test Significance Calculator
Is your test result real?
Reviewed by the Calculation Machine team · Updated 24 October 2026 · How we check our maths
Your values
ⓘ Live results
- Significant?
- Yes at 95% confidence
- Control rate
- 6%
- Variant rate
- 7.2%
- Relative lift
- 20.1%
Quick answer
An A/B test result is statistically significant when the gap between the two conversion rates is too large to be explained by random chance at your chosen confidence level. The standard check is a two-proportion z-test: if |z| is 1.96 or more, the result is significant at 95%. A control converting 312 of 5,200 visitors (6.0%) against a variant converting 371 of 5,150 (7.2%) gives z ≈ 2.47, so the 20.1% lift is significant.
What the A/B Test Significance calculator does
This calculator compares the conversion rates of two versions of a page, email or ad and tells you whether the difference is statistically significant at 95% confidence (two-sided). It also shows each version's conversion rate and the relative lift of the variant over the control.
Significance answers one narrow question: how surprising would this gap be if the two versions really performed the same? It does not tell you the size of the true effect, and it is only valid if you fixed the sample size before starting and did not stop the test early because the numbers looked good. Work out how many visitors you need in advance with the sample size calculator.
The test assumes each visitor is counted once, sees only one version, and converts at most once. Revenue per visitor and average order value need a different test (a t-test on the values), because they are not simple yes/no outcomes. For single-page rates, see the conversion rate calculator.
The formula
p₁ = c₁ ÷ n₁, p₂ = c₂ ÷ n₂ p = (c₁ + c₂) ÷ (n₁ + n₂) z = (p₂ − p₁) ÷ √( p(1 − p)(1/n₁ + 1/n₂) ) Significant at 95% if |z| ≥ 1.96
| Symbol | Meaning | Unit |
|---|---|---|
| n₁, n₂ | Visitors (or users) who saw the control and the variant | count |
| c₁, c₂ | Conversions for the control and the variant | count |
| p | Pooled conversion rate across both groups | proportion |
| z | Test statistic: the difference in standard errors | — |
| 1.96 | Critical value for a two-sided test at 95% confidence (α = 0.05) | — |
This is the pooled two-proportion z-test found in introductory statistics texts such as OpenIntro Statistics and NIST's Engineering Statistics Handbook. It relies on the normal approximation, which is reasonable when each group has at least about 10 conversions and 10 non-conversions. With very small counts, use Fisher's exact test instead.
Worked examples
Checkout redesign with a clear win
An online store tests a shorter checkout: 5,200 visitors saw the old one (312 orders) and 5,150 saw the new one (371 orders).
Inputs
- Control visitors5200
- Control conversions312
- Variant visitors5150
- Variant conversions371
Result
- Significant?Yes at 95% confidence
- Control rate6%
- Variant rate7.2%
- Relative lift20.1%
The variant converts at 7.2% versus 6%, a 20.1% lift, and z ≈ 2.47 (p ≈ 0.014) clears the 95% threshold. The true lift is uncertain, though — the 95% interval on the difference is wide, so expect the long-run gain to be somewhat smaller.
Big lift, too little data
A SaaS signup page test after one week: control 48 signups from 1,200 visitors, variant 61 from 1,180.
Inputs
- Control visitors1200
- Control conversions48
- Variant visitors1180
- Variant conversions61
Result
- Significant?Not yet significant
- Control rate4%
- Variant rate5.17%
- Relative lift29.2%
A 29.2% lift sounds decisive, but z ≈ 1.37 (p ≈ 0.17) is nowhere near significant. Detecting a lift from 4% to 5% reliably needs roughly 6,700 visitors per version, so keep the test running to the planned sample size.
Large test where the variant is worse
A retailer tests a new homepage banner across 80,000 visitors: control 1,240 orders from 40,000, variant 1,150 from 40,200.
Inputs
- Control visitors40000
- Control conversions1240
- Variant visitors40200
- Variant conversions1150
Result
- Significant?Yes at 95% confidence
- Control rate3.1%
- Variant rate2.86%
- Relative lift-7.7%
The variant converts at 2.86% against 3.1%, a 7.7% drop, and z ≈ −1.99 is just past the threshold. Significant results can go either way; here the test has protected revenue by showing the change should not ship.
Visitors needed per variant
Sample sizes below use the standard formula for comparing two proportions at 95% confidence and 80% power. Small relative lifts on low baseline rates need surprisingly large samples, which is why many low-traffic sites cannot test subtle changes.
| Baseline rate | Variant rate | Relative lift | Visitors per variant |
|---|---|---|---|
| 2% | 2.5% | 25% | about 13,800 |
| 3% | 3.3% | 10% | about 53,000 |
| 4% | 5% | 25% | about 6,700 |
| 5% | 6% | 20% | about 8,200 |
| 10% | 11% | 10% | about 14,800 |
Mistakes that create false winners
- Peeking and stopping early: checking every day and stopping the first time the result is significant can push the real false-positive rate far above 5%. Decide the sample size up front, or use a sequential testing method designed for repeated looks.
- Running too short: run for whole weeks so weekday and weekend visitors are both represented, even if you reach the sample size sooner.
- Testing many variants or metrics: with 20 independent comparisons at 95% confidence, one is expected to be "significant" by chance. Pick a primary metric in advance.
- Sample ratio mismatch: if a 50/50 split delivers noticeably uneven visitor counts, the randomization or tracking is broken and the result should not be trusted.
- Novelty effects: returning users may click a new design simply because it is new. Check whether the lift holds in the last week of the test.
Reading p-values and confidence correctly
A p-value of 0.03 means that if the two versions truly converted at the same rate, a gap this large would appear about 3% of the time. It does not mean there is a 97% chance the variant is better. Significance also says nothing about whether the lift is large enough to matter commercially — check the confidence interval on the difference and ask whether its lower end would still justify the change.
How to use the A/B Test Significance Calculator
- 1Enter the number of visitors who saw the control (version A) and how many of them converted.
- 2Enter the same two numbers for the variant (version B), from the same date range.
- 3Use unique visitors or users if your testing tool reports them, so repeat visits are not double counted.
- 4Read whether the result is significant at 95% confidence, then the two conversion rates and the relative lift.
Frequently asked questions
What does statistically significant mean in an A/B test?
It means the difference between versions is unlikely to be due to chance alone, at the confidence level you chose. At 95% confidence you accept a 5% risk of declaring a winner when there is no real difference.
How long should I run an A/B test?
Until you reach the sample size you calculated beforehand, and for at least one or two full weeks to cover weekly traffic patterns. Stopping as soon as a result looks significant inflates false positives.
What is a good p-value for an A/B test?
Most teams use 0.05 (95% confidence) as the threshold. Use 0.01 for changes that are expensive or hard to reverse, and accept 0.10 only for low-risk tweaks.
Is relative lift the same as the difference in conversion rate?
No. Going from 6% to 7.2% is a 1.2 percentage-point difference but a 20% relative lift. Report both so stakeholders are not misled by the larger-sounding number.
Can I use this for more than two variants?
You can compare each variant with the control one at a time, but the more comparisons you make the more likely one is a false positive. Apply a correction such as Bonferroni (divide 0.05 by the number of comparisons) or use a tool built for multivariate tests.
References
- OpenIntro Statistics (difference of two proportions) — OpenIntro
- NIST/SEMATECH e-Handbook of Statistical Methods — National Institute of Standards and Technology
- Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing — Cambridge University Press (Kohavi, Tang and Xu)
About this calculator
- Built and reviewed by the Calculation Machine team · How we build and test calculators
- Last reviewed: October 24, 2026
- Report an error