Enter AB Test Inputs
Example Data Table
| Variant | Visitors | Conversions | Conversion Rate | Revenue per Conversion | Notes |
|---|---|---|---|---|---|
| A | 1000 | 120 | 12.00% | $42.00 | Original checkout design |
| B | 980 | 150 | 15.31% | $41.50 | Updated button and headline |
Formula Used
A chi-square AB test compares observed conversions and non-conversions against the counts expected if both variants truly convert at the same rate.
For a 2×2 table, the core statistic is: χ² = Σ ((O − E)² / E), where O is the observed count and E is the expected count.
In this calculator, the table is: [A conversions, A non-conversions, B conversions, B non-conversions]. Expected counts are based on the pooled conversion rate across both variants.
Conversion rates are: Rate A = Conversions A / Visitors A and Rate B = Conversions B / Visitors B.
Absolute lift is: Rate B − Rate A. Relative lift is: (Rate B − Rate A) / Rate A.
For degree of freedom 1, the p-value is the right-tail probability of the chi-square distribution. When selected, Yates continuity correction slightly reduces the statistic for conservative inference in small samples.
The confidence interval shown here uses the unpooled standard error for the difference in conversion rates. Effect size is reported with phi: φ = √(χ² / N).
How to Use This Calculator
- Enter visitor counts for Variant A and Variant B.
- Enter conversions recorded for each variant.
- Choose your alpha level, such as 0.05.
- Set the confidence level for the lift interval.
- Optionally enable Yates correction for conservative testing.
- Click the calculate button to generate metrics and charts.
- Review rates, lift, p-value, confidence interval, and effect size.
- Use the CSV or PDF buttons to save the analysis.
FAQs
1. What does this calculator test?
It tests whether the difference between two conversion rates is likely due to chance. The method compares observed outcomes against expected counts under a no-difference assumption.
2. When should I use a chi-square AB test?
Use it when both variants have count data, like visitors and conversions. It works well for binary outcomes such as signups, purchases, clicks, or completed forms.
3. What is a p-value here?
The p-value estimates how surprising your observed difference would be if both variants truly performed the same. Smaller values provide stronger evidence against equal conversion rates.
4. What does statistical significance mean?
It means the p-value is below your chosen alpha threshold. That suggests the measured conversion difference is unlikely to be explained by random sampling alone.
5. Why report effect size and lift too?
Significance only indicates evidence, not business value. Lift shows practical improvement, while effect size helps you judge whether the difference is tiny, moderate, or meaningful.
6. What is Yates continuity correction?
It is a conservative adjustment for 2×2 chi-square tables. Analysts sometimes use it when sample counts are small and they want to reduce overstatement of significance.
7. Can I use this for very small samples?
Be careful with very small expected counts. If expected cells fall below common thresholds, an exact method such as Fisher’s exact test may be more appropriate.
8. Does a winning variant guarantee better future results?
No. Results depend on traffic quality, seasonality, instrumentation, and sampling variation. Validate stable gains with enough data and consider repeating the experiment.