liveBook · Manning

Bayesian A/B testing

As explained in Timeless Algorithms, The Foundational Ideas

Bayesian A/B testing compares alternatives by updating probability distributions about their performance using observed data.

Bayesian A/B testing compares alternatives by updating prior beliefs about their performance as data arrives. Observed clicks, purchases, or conversions produce posterior distributions that represent current beliefs about each option’s performance. section 2.4.1

The comparison focuses on what the evidence says about the alternatives:

  • Estimate the probability that one option outperforms another and the likely magnitude of its improvement. section 2.4.1
  • Unlike a traditional hypothesis test, ask which option is more likely to be better rather than whether evidence is statistically significant against a null hypothesis of no difference. section 2.4.1

Choosing a call-to-action color

A product team begins with equal prior beliefs about a blue and an orange button. Click data from users update the beliefs into posterior distributions for the two designs’ true click-through rates. section 2.4.1

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Defines prior and posterior distributions, then interprets comparative probabilities and likely improvement magnitude.
Examples
Uses a blue-versus-orange call-to-action button and observed user clicks.
What you can do after
Evaluate which alternative is more likely to perform better and estimate the likely magnitude of improvement.
How it connects
Connects Bayes’ theorem and posterior distributions to contrasts with null-hypothesis testing.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.