Calculator inputs
The page uses a single-column content flow, while the form fields adapt to 3, 2, and 1 columns.
Example data table
Use this sample to test the calculator quickly and review the output structure.
| Variant | Visitors | Conversions | Prior α | Prior β | Observed conversion rate |
|---|---|---|---|---|---|
| A | 10,000 | 520 | 1.00 | 1.00 | 5.20% |
| B | 9,800 | 575 | 1.00 | 1.00 | 5.87% |
Formula used
1) Posterior update
For each variant, the conversion rate is modeled with a Beta prior. After observing conversions, the posterior becomes: Beta(α + conversions, β + failures).
2) Posterior mean
Posterior mean conversion rate = (α + conversions) / (α + β + visitors).
3) Winning probability
Probability(B > A) is estimated by Monte Carlo simulation. The calculator repeatedly samples from both posterior distributions, then counts how often Variant B exceeds Variant A.
4) Uplift and regret
Absolute uplift = sampled rate of B − sampled rate of A. Relative uplift = (posterior mean B − posterior mean A) / posterior mean A. Expected loss measures the average missed lift from choosing the weaker option.
How to use this calculator
- Enter visitor and conversion counts for both variants.
- Set prior alpha and beta values for each variant.
- Choose the simulation count and credible level.
- Set the decision threshold for selecting a winner.
- Optionally add revenue per conversion for business impact.
- Submit the form to view posterior curves and summary metrics.
- Review winning probability, uplift interval, and expected loss.
- Download the result as CSV or PDF for reporting.
FAQs
1) What does Bayesian A/B testing measure?
It estimates the probability that one variant performs better than another. Instead of only testing significance, it gives a direct probability statement and a range of believable conversion rates.
2) Why are alpha and beta inputs included?
They define the Beta prior for each variant. Higher values add stronger prior belief. Using 1 and 1 creates a uniform prior, which is a common neutral starting point.
3) What does the winning probability mean?
It is the estimated chance that Variant B has a higher true conversion rate than Variant A, based on the observed data and chosen priors.
4) What is a credible interval?
A credible interval is the Bayesian uncertainty range for a conversion rate or uplift. It shows where the true value is likely to lie given the model and data.
5) Why use simulations instead of a closed-form shortcut?
Simulations make it easy to estimate winning probability, uplift distributions, and regret in one framework. They are flexible and practical for reporting richer decision metrics.
6) What does expected loss tell me?
Expected loss estimates how much conversion rate you may give up by choosing a variant that later turns out to be weaker. Lower expected loss supports safer decisions.
7) Can I use this for revenue decisions?
Yes. Enter revenue per conversion to translate uplift into expected revenue per 1,000 visitors. This helps connect statistical evidence to practical business impact.
8) How many simulations should I use?
Around 20,000 to 50,000 is usually a strong balance between stability and speed. Larger counts reduce Monte Carlo noise but require more processing time.