Test Results
Interpretation
Run the calculator to see the decision summary.
Posterior Simulation Snapshot
Enter Experiment Inputs
Example Data Table
| Variant | Visitors | Conversions | Observed Conversion Rate | Prior Alpha | Prior Beta |
|---|---|---|---|---|---|
| A | 5,000 | 420 | 8.40% | 1 | 1 |
| B | 5,100 | 468 | 9.18% | 1 | 1 |
This sample shows Variant B leading on observed conversion rate. The calculator then estimates how likely that advantage remains true after Bayesian updating.
Formula Used
This calculator uses a Beta-Binomial Bayesian model for conversion testing. Each variant starts with a prior:
Prior: Beta(α, β)
After observing conversions, the posterior distribution becomes:
Posterior: Beta(α + conversions, β + failures)
where failures = visitors - conversions.
- Posterior mean: (α + conversions) / (α + β + visitors)
- Expected uplift of B: ((MeanB - MeanA) / MeanA) × 100
- Probability B beats A: simulated share of posterior draws where B > A
- Expected loss if choosing B: average positive regret when A outperforms B
- 95% credible interval: 2.5th and 97.5th percentiles of posterior draws
- Profit estimate: incremental conversions × revenue per conversion - incremental visitors × cost per visitor
Monte Carlo simulation approximates the posterior comparison and practical decision metrics.
How to Use This Calculator
- Enter visitors and conversions for Variant A and Variant B.
- Set prior alpha and beta values for each variant.
- Choose the simulation count for posterior sampling precision.
- Enter a practical lift target percentage for decision support.
- Add revenue per conversion and cost per visitor if profit matters.
- Click Calculate Bayesian Test.
- Review win probabilities, credible intervals, uplift, expected loss, and profit estimate.
- Use CSV or PDF export to save results.
Frequently Asked Questions
1. What does Probability B beats A mean?
It shows the posterior chance that Variant B has a higher true conversion rate than Variant A after combining prior beliefs and observed data.
2. Why use Bayesian testing instead of a fixed p-value?
Bayesian analysis gives direct probability statements, supports prior knowledge, and helps decision-making through uplift and regret estimates rather than threshold-only significance rules.
3. What prior should I choose?
Use alpha = 1 and beta = 1 for a neutral starting point. Use stronger priors only when you have trusted historical conversion evidence.
4. What is expected loss?
Expected loss measures regret. It estimates how much conversion rate you may give up on average if you choose one variant and the other is better.
5. What is a credible interval?
A credible interval gives a probability-based range for the true conversion rate. It is directly interpretable under the Bayesian model assumptions.
6. Does this calculator work for low-traffic tests?
Yes. Bayesian methods often remain useful with smaller samples, though uncertainty stays wider and decisions should be more cautious.
7. Why include a practical lift target?
A small probability advantage may not justify deployment. The lift target checks whether B is likely to exceed a meaningful business improvement threshold.
8. Can I export the results?
Yes. Use the CSV button for structured data and the PDF button for a clean report of the displayed Bayesian test summary.