liveBook · Manning

Prior predictive check

As explained in Grokking Bayes

A prior predictive check is a diagnostic procedure that assesses whether data generated from a model's prior assumptions appear realistic before fitting the model to observed data.

Before surveying 100 people about tea and coffee, what results would your model consider possible? A prior predictive check answers that question by generating hypothetical data before looking at the survey results. If those results clash with what you know about the situation, reconsider the model’s assumptions before fitting it. section 4.2 section 4.3

The check follows the model all the way to observable data:

  • Draw possible values of an unknown quantity from its prior. section 4.3
  • For each draw, generate a possible dataset using the model’s rule for observations. section 4.3
  • Inspect the simulated datasets and ask whether their range and patterns make sense before seeing actual outcomes. section 4.1 section 4.3

Tea preferences

Let p be the proportion of people who prefer tea, and k the number of tea drinkers in a survey of 100. Draw p from its prior, then draw k for each p. A uniform prior on p makes every count from 0 to 100 equally likely. That passes the check if any count seems reasonable, but fails if you already believe most people prefer coffee. A strongly skewed prior such as Beta(0.1, 10) instead favors counts near zero; it fails if counts that low seem implausible. Inspect k, not just p: the question is what data the assumptions generate. section 4.3

from scipy.stats import beta, binom
prior_p_samples = beta.rvs(
    prior_a, 
    prior_b, 
    size=10_000,
)
prior_k_samples = binom.rvs(
    100,                     #A
    prior_p_samples,         #B
)

The Python example draws 10,000 values of p with SciPy’s beta.rvs, then passes those values to binom.rvs to generate possible survey counts. section 4.3

House prices

For a house-price model, draw possible slopes from the slope’s prior, then use chosen house areas and simulated noise to generate prices. Areas can even come from the training set, provided their actual prices are not used in this check. Plotting the generated price distributions across areas makes the prior’s implications easier to judge. An adjusted slope prior produces a different set of predictions. section 4.4

A prior predictive check tests assumptions, not whether a model has learned from observed data. After fitting, a posterior predictive check asks the different question of whether simulated data can reproduce the observations. section 4.0 section 4.12

How Grokking Bayes covers it

How deep it goes
Works through sampling, simulated datasets, and interpretation for two models; stops short of a formal pass-or-fail threshold.
Examples
Uses tea-drinker counts under two Beta priors and simulated house prices across chosen floor areas.
What it assumes
Assumes familiarity with priors and likelihoods, while introducing distribution sampling as the practical basis for the check.
What you can do after
Apply it via SciPy sampling to identify implausible prior-induced data before fitting a model.
How it connects
Connects prior distributions and likelihoods to simulated observations, and contrasts checks before fitting with posterior predictive checks.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Bayes. The section links point to the parts of the book it draws on.