Before surveying 100 people about tea and coffee, what results would your model consider possible? A prior predictive check answers that question by generating hypothetical data before looking at the survey results. If those results clash with what you know about the situation, reconsider the model’s assumptions before fitting it. section 4.2 section 4.3
The check follows the model all the way to observable data:
- Draw possible values of an unknown quantity from its prior. section 4.3
- For each draw, generate a possible dataset using the model’s rule for observations. section 4.3
- Inspect the simulated datasets and ask whether their range and patterns make sense before seeing actual outcomes. section 4.1 section 4.3
Tea preferences
Let p be the proportion of people who prefer tea, and k the number of tea drinkers in a survey of 100. Draw p from its prior, then draw k for each p. A uniform prior on p makes every count from 0 to 100 equally likely. That passes the check if any count seems reasonable, but fails if you already believe most people prefer coffee. A strongly skewed prior such as Beta(0.1, 10) instead favors counts near zero; it fails if counts that low seem implausible. Inspect k, not just p: the question is what data the assumptions generate. section 4.3
from scipy.stats import beta, binom
prior_p_samples = beta.rvs(
prior_a,
prior_b,
size=10_000,
)
prior_k_samples = binom.rvs(
100, #A
prior_p_samples, #B
)
The Python example draws 10,000 values of p with SciPy’s beta.rvs, then passes those values to binom.rvs to generate possible survey counts. section 4.3
House prices
For a house-price model, draw possible slopes from the slope’s prior, then use chosen house areas and simulated noise to generate prices. Areas can even come from the training set, provided their actual prices are not used in this check. Plotting the generated price distributions across areas makes the prior’s implications easier to judge. An adjusted slope prior produces a different set of predictions. section 4.4
A prior predictive check tests assumptions, not whether a model has learned from observed data. After fitting, a posterior predictive check asks the different question of whether simulated data can reproduce the observations. section 4.0 section 4.12
