11 Bayesian hypothesis testing: Asking the right questions when hypothesizing
This chapter covers
- The Bayesian approach to hypothesis testing
- The Bayes factor as an alternative to 𝑝-values
- Moving beyond hypothesis testing
In classical (frequentist) statistics, hypothesis testing is built around the famous 𝑝-value, a statistical measure widely used to determine whether the results of an experiment are meaningful or just occurring by chance. We start with a null hypothesis, compute how surprising the data would be if that hypothesis were true, and reject the null if the result crosses a predetermined threshold.
This framework is widely used, but it often leaves important questions unanswered. For example, a 𝑝-value tells us how unusual the data would be under the null hypothesis (for example, what’s the chance that you see this many clicks on a new website that’s no better than the current website), but it doesn’t tell us how likely that hypothesis, or any other, is. A 𝑝-value below the predetermined threshold also doesn’t directly inform our decision making; that is, we can’t directly answer questions about which action to take given our knowledge about a problem, such as whether the new website is worth deploying.