Bayesian A/B testing compares alternatives by updating prior beliefs about their performance as data arrives. Observed clicks, purchases, or conversions produce posterior distributions that represent current beliefs about each option’s performance. section 2.4.1
The comparison focuses on what the evidence says about the alternatives:
- Estimate the probability that one option outperforms another and the likely magnitude of its improvement. section 2.4.1
- Unlike a traditional hypothesis test, ask which option is more likely to be better rather than whether evidence is statistically significant against a null hypothesis of no difference. section 2.4.1
Choosing a call-to-action color
A product team begins with equal prior beliefs about a blue and an orange button. Click data from users update the beliefs into posterior distributions for the two designs’ true click-through rates. section 2.4.1
