chapter four

4 Model checking and comparison: Evaluating and picking out the best Bayesian model

 

This chapter covers

  • Prior predictive checks for sanity-checking assumptions
  • Posterior predictive checks for accessing learned models
  • Marginal likelihoods and Bayes factors for comparing different models against one another

A Bayesian model combines a prior distribution and a likelihood function to yield a posterior distribution to quantify our updated knowledge about a phenomenon of interest. But there’s a question we haven’t explicitly considered: How do we know if our model is any good? With a model with multiple components, many things can go wrong.

It's tempting to assume from the examples from previous chapters that our job is done if the math works out and the posterior looks reasonable, but this is not true. Like any other statistical and machine learning model, a Bayesian model can be wrong in subtle and important ways. For example, maybe our prior doesn’t truly reflect our knowledge, or our likelihood is not suitable to capture the patterns in the data.

Sampling from a probability distribution

Prior predictive checks: Do our assumptions generate realistic data?

Prior predictive checks for the beverage preference model

Prior predictive checks for the house price model

Posterior predictive checks: Can our model reproduce the data?

Posterior predictive checks for the beverage preference model

Posterior predictive checks for the house price model

Comparing Bayesian models

The marginal likelihood

The Bayes factor

The Bayes factor as a Bayesian Occam’s razor

Summary