Understanding Bayesian A/B Testing
Bayesian A/B testing treats each conversion rate as an unknown value. It updates belief as real observations arrive. The method starts with a prior beta distribution. The prior can be flat, cautious, or based on past campaigns. After visits and conversions are entered, the calculator builds posterior beta distributions for both variants. These posteriors describe many likely true rates, not only one point estimate.
Why Posterior Probability Matters
Traditional tests often focus on a p value. Bayesian analysis asks a more direct question. It estimates the chance that variant B is better than variant A. It can also estimate the chance that A is better. This is useful for marketing pages, emails, pricing tests, product flows, and signup funnels. Teams can judge evidence in business terms. They can compare probability, lift, and possible regret before choosing a winner.
Risk, Loss, and Practical Lift
A variant may have a high probability of winning, yet still create small value. That is why practical lift matters. The calculator lets you enter a minimum useful lift. It also estimates expected loss per visitor. Expected loss shows the average penalty from choosing a weaker variant. Revenue per conversion turns rate differences into money. This helps teams avoid overreacting to tiny gains or noisy traffic.
Using Priors Responsibly
Priors should reflect honest knowledge. A flat prior, such as one and one, is common for new tests. Stronger priors may help when many similar tests already exist. However, very strong priors can hide new evidence. Use equal priors for fair comparisons unless you have a clear reason. Check sample size, traffic balance, and conversion tracking before trusting any result. Bayesian results are easy to read, but data quality still matters.
Decision Guidance
Use the recommendation as a guide, not an automatic rule. Launch a winner when posterior probability is high, loss is low, and lift is meaningful. Keep testing when the result is uncertain. Stop early only when the business risk is acceptable. Good decisions combine statistics, product judgment, and customer impact.
Document each test rule before launch. Record the prior choice, decision threshold, stopping plan, and main metric. This makes later reviews cleaner and reduces biased interpretation during analysis afterward.