Skip to content
MasterMath

A/B Test Significance Calculator

Whether the gap between two versions of a page is real or just noise. It gives the p-value, the relative lift and the interval for the difference, which is what really deserves attention.

Conclusion

—

Conclusion—
p-value—
Relative lift of B—
Absolute difference—
Interval for the difference—
Observed confidence—
VersionVisitsConversionsRate

How this was worked out

    The formula

    z = (p_B − p_A) / √( p̂(1 − p̂)(1/n_A + 1/n_B) )

    What it means

    An A/B test compares two conversion rates. The question is not which number is higher — one always will be — but whether that gap is explained by the chance of who landed on each version. The test works out how many standard errors separate the two rates and turns that into a probability: if it is very small, chance is a poor explanation.

    How to work it out by hand

    1. Work out the conversion rate of each version
    2. Work out the pooled proportion and its standard error
    3. Divide the difference in rates by that error: that is the z statistic
    4. Turn z into a p-value and compare it against the level you set

    What is worth knowing

    Checking the test every day and stopping the moment it turns significant is the most common way to fool yourself: with enough looks, almost any test crosses the threshold at some point even with no real difference. Fix the sample size in advance and wait. And the relative lift, the figure that gets taken into the meeting, is the least stable of all: an interval for the difference running from 0.1 % to 3 % can correspond to a relative lift of anywhere between 2 % and 60 %.

    Frequently asked questions

    When can I stop the test?

    When you reach the sample size you fixed before starting. Stopping as soon as it turns significant inflates false positives enormously.

    What does the p-value mean here?

    The probability of seeing a gap at least this large if the two versions were really identical.

    Why is the interval more important than the p-value?

    Because it says how big the improvement might be, not just whether one exists. A significant result with an interval from 0.1 % to 3 % is nearly useless for deciding.

    Relative or absolute lift?

    The absolute one drives the maths; the relative one impresses. Quote both and do not trust the second on its own.

    How many visits do I need?

    It depends on the baseline rate and the improvement you want to detect. The sample size calculator gives the order of magnitude.