Calculator inputs
Use traffic, conversions, priors, revenue per conversion, and business thresholds. Results appear above this form after submission.
Example data table
| Variant | Visitors | Conversions | Observed Rate | Revenue per Conversion | Suggested Prior |
|---|---|---|---|---|---|
| A | 18,000 | 684 | 3.80% | $52.00 | Beta(1,1) |
| B | 17,500 | 742 | 4.24% | $57.00 | Beta(1,1) |
This sample mimics a marketing test where both conversion probability and order value matter. It is already loaded into the form.
Formula used
Posterior update: If a variant has conversions x from n visitors and prior Beta(α, β), then the posterior becomes Beta(α + x, β + n − x).
Posterior mean conversion rate: θ̄ = αpost / (αpost + βpost).
Observed uplift: (CRB − CRA) / CRA.
Expected revenue per visitor: conversion rate × revenue per conversion.
Chance to beat control: estimated by Monte Carlo sampling, using the share of sampled outcomes where B exceeds A.
Expected loss: average downside from choosing a variant when sampled reality favors the other variant.
How to use this calculator
- Enter visitors and conversions for both variants.
- Add revenue per conversion if the business outcome matters.
- Set prior alpha and beta values to reflect earlier belief.
- Choose a certainty threshold for decision confidence.
- Set a minimum practical uplift to avoid tiny wins.
- Select the credible interval level and sample size.
- Press the calculate button to generate the result summary.
- Review the posterior chart, risk metrics, and recommendation.
- Export the analysis as CSV or PDF if needed.
FAQs
1) What does Bayesian A/B testing add beyond a standard lift calculator?
It estimates full probability distributions, not only point estimates. That helps you judge uncertainty, expected loss, decision confidence, and the chance that a variant truly beats control.
2) What are alpha and beta priors?
They describe your belief about conversion rate before current data arrives. A Beta(1,1) prior is uniform and neutral, while larger values add stronger prior influence.
3) When should I use revenue per conversion?
Use it when conversion value differs across variants. A version with a slightly lower rate can still be the better business choice if it creates more revenue per visitor.
4) What is expected loss?
Expected loss measures the average downside of choosing one variant when posterior samples favor the other. Lower expected loss usually means a safer decision under uncertainty.
5) Why include a practical uplift threshold?
Some wins are statistically plausible but too small to matter. A practical uplift filter focuses decisions on outcomes large enough to justify implementation cost or budget movement.
6) How many Monte Carlo samples should I use?
Higher sample counts give smoother probability estimates but take longer. Around 8,000 to 12,000 usually balances stability and speed for marketing experiments.
7) Can this calculator handle low-traffic experiments?
Yes. Bayesian methods are often useful with smaller samples because they express uncertainty directly. Still, extremely sparse data will produce wider intervals and weaker conclusions.
8) Should I always launch B if its probability is higher?
Not always. You should also inspect expected loss, revenue impact, credible intervals, and your minimum practical uplift. A small edge may be too uncertain or commercially unimportant.