Analytics · Runs in your browser
A/B test significance calculator
Type the visitors and conversions for A and B. The p-value, confidence interval and lift update as you type. Free, unlimited, no account, and nothing is uploaded.
Your test data
AControl
BVariant
Choose the direction before you look at the numbers. Switching to one-sided after seeing the result halves the p-value and is not a valid test.
Result
Test plan
Visitors you need
Halving the effect you want to detect roughly quadruples the traffic you need.
What you get
Check a finished test
- Conversion rate of A and B
- Difference in percentage points and relative lift
- z statistic and p-value, two-sided or one-sided
- Confidence interval on the difference at 90, 95 or 99%
- A written reading of the result, with a warning when a group has fewer than 10 conversions or non-conversions
Plan the sample size
- Visitors needed per variant and in total
- Expected conversions in each group
- The variant rate the test is sized for
- Run time in days and weeks at your daily traffic
- Power of 70, 80, 90 or 95%, and a relative or absolute effect
Each value has its own Copy button for pasting into a report or ticket.
How to read the result
The method
The calculator treats each conversion rate as a binomial proportion and runs a z-test on the difference between A and B. The test statistic uses the pooled rate; the confidence interval keeps the two rates separate. This normal approximation works once each group has about 10 conversions and 10 non-conversions. Below that, treat the output as a rough guide.
What a p-value does not say
A p-value of 0.03 means that if A and B really performed the same, a gap at least this large would turn up about 3% of the time. It does not mean a 97% chance that B is better, and it says nothing about how much better. The confidence interval answers that.
Do not stop early
Fix the sample size before the test and read the result once, when it is reached. Checking every day and stopping the first time p drops below 0.05 pushes the real false-positive rate well above 5%.
Significant is not the same as worth shipping
With enough traffic, a very small lift becomes significant. Look at the low end of the confidence interval. If it runs from +0.1 to +3 points, you do not yet know whether the change pays for itself. That decision stays yours.
What it does not cover
It compares two groups at a time and uses a fixed-horizon frequentist test. It does not run Bayesian or sequential analysis, and it does not handle revenue per visitor or other continuous metrics.
Questions
How do I work out the sample size before a test?
Switch to Plan sample size. Enter your baseline conversion rate, the smallest effect you want to detect, the significance level and the power (80% is a common choice). You get the visitors each variant needs and the total. Add your daily traffic to see how many days the test has to run. With fewer visitors, a real effect can go unnoticed.
What does a significant result mean?
It means the gap between A and B would be unlikely if both versions performed the same. At a significance level of 0.05, a p-value below 0.05 counts as significant. It does not tell you how large the effect is or whether it matters for your business, so read it together with the confidence interval and the lift.
What is statistical power and why does it matter?
Power is the chance that a test detects an effect of at least the size you planned for, if that effect is real. At 80% power, one test in five misses it. A test with low power can run for weeks, show no significant result, and lead you to drop a change that works. You set power when you plan the sample size.
Can I compare more than two variants?
The calculator compares two groups at a time. With three or more variants, compare each against the control and use a stricter significance level, such as 0.01, because every extra comparison raises the chance of a false positive.
Is there a limit, and is my data uploaded?
There is no limit and no account. The calculator runs in your browser, so the numbers you type are not sent to a server and are not stored. Copy the values you want to keep before you close the page.