Test Power Calculator for Marketing Experiments

Calculator

Use conversion rates as decimals. Example: 0.08 equals 8%.

Download CSV Download PDF

Calculation mode

Test tails

Use one-tailed only with a strong directional hypothesis.

Significance level (alpha)

Enter alpha between 0.0001 and 0.5.

Baseline conversion (p1)

Example: 0.08 means 8%.

Enter p1 between 0 and 1.

Variant conversion (p2)

Used in power and sample size modes.

Enter p2 between 0 and 1.

Target power

Common targets: 0.80 or 0.90.

Enter target power between 0.01 and 0.999.

Control sample size (n1)

Used in power and MDE modes.

Enter n1 ≥ 2.

Allocation ratio (n2 / n1)

1.0 means equal split. n2 is derived.

Enter ratio between 0.1 and 10.

MDE direction

Only used in MDE mode.

Clear

Example Data Table

These examples show typical marketing A/B test settings and outputs.

Scenario	p1	p2	Alpha	Tails	n1	Ratio	Output
Landing page headline test	8%	10%	0.05	Two	10,000	1.0	Power ≈ 80–90% (typical)
Email subject line test	2.5%	3.0%	0.05	Two	25,000	1.0	Power improves as sample grows
Paid search bid strategy test	5%	5.6%	0.10	One	15,000	1.5	Directional test with uneven split

Formula Used

This tool uses a two-proportion z-test approximation for conversion rates.

p1 is the control conversion rate, p2 is the variant conversion rate.
Absolute lift is Δ = p2 − p1.
Standard error under the alternative is SE = √( p1(1−p1)/n1 + p2(1−p2)/n2 ).
Signal is μ = Δ / SE.
Critical value is zα from the standard normal distribution.

For a two-tailed test, power is computed as the probability of exceeding the critical z-threshold in either direction. For a one-tailed test, power uses the directional threshold.

Note: This is a fast planning approximation. For low traffic or extreme rates, consider exact methods.

How to Use This Calculator

Enter the baseline conversion rate from your recent data.
Choose a mode: power, sample size, or minimum detectable effect.
Set alpha and tails based on your decision policy.
Enter sample size and allocation ratio when needed.
Press Submit. The result appears above the form.
Download CSV or PDF for reporting and sharing.

FAQs

1) What does “power” mean for a marketing A/B test?

Power is the chance your test detects a real lift when it exists. Higher power reduces false negatives, helping you avoid missing valuable improvements in campaigns.

2) Should I use one-tailed or two-tailed testing?

Two-tailed is safer for most teams because it detects both lifts and drops. Use one-tailed only when you will never act on a negative result and your hypothesis is strictly directional.

3) What is MDE and why does it matter?

MDE is the smallest absolute change you can reliably detect at your alpha and power. Smaller MDE needs more traffic, but it allows you to detect subtle creative or UX gains.

4) How do I pick an alpha level?

Alpha is your tolerance for false positives. Many marketing teams use 0.05. If decisions are expensive or risky, you can lower alpha, but you will usually need more sample size.

5) Why does an uneven split change results?

Uneven allocation increases variance for the smaller group, which can reduce power for a fixed total sample. You might still use it to limit risk on a new experience or offer.

6) Can I use this for revenue per user instead of conversion?

This calculator is built for binary outcomes, like conversion. For continuous metrics such as revenue per user, you typically use a t-test power model with a mean difference and standard deviation.

7) Why might my observed results differ from the estimate?

Real data has seasonality, targeting changes, and tracking noise. The normal approximation also simplifies behavior at extreme rates. Treat outputs as planning guidance and validate with your analytics workflow.