A prior probability, written p(θ), is a distribution over unknown parameters before the current training data is considered. It encodes a belief or constraint that is independent of X. Bayes’ rule combines it with the likelihood p(X|θ): the posterior p(θ|X) favors parameter values supported both by the data and by the prior. Priors matter most when data is scarce; as data grows, the likelihood increasingly determines the estimate. section 6.6.1 section 6.6.3
A common parameter prior prefers small values: p(θ) ∝ e^(−||θ||²). Large-magnitude parameter vectors then receive lower prior probability. This expresses a preference for simpler parameter settings and turns Bayesian parameter estimation into a practical constraint rather than an unsupported guess. section 6.6.1 section 6.10
The prior enters estimation in two closely related ways:
- MLE maximizes p(X|θ) and uses only the observed data. section 6.10
- MAP maximizes p(X|θ)p(θ), equivalently adding the prior as a regularizing influence on likelihood-based estimation. section 6.6.3
- Fully Bayesian estimation keeps a posterior distribution over parameters, allowing uncertainty and prior strength to remain explicit. section 13.11
Gaussian mean with limited data
For Gaussian data with known variance and unknown mean μ, model μ itself with a Gaussian prior centered at μ₀ and having variance σ₀. The posterior mean combines the sample-based MLE mean with μ₀. With few observations, the posterior stays close to the prior; with many observations, it approaches the MLE. A wider, weaker prior is easier for data to overwhelm, while a narrower, stronger prior resists it. section 13.3 section 13.11
| In MAP parameter estimation, we look for parameters θ that maximize p(X|θ)p(θ).
|
MAP makes the division of labor explicit: likelihood fits the data, while the prior promotes predefined parameter properties such as smaller magnitudes. section 6.6.3
The same idea applies beyond model parameters. In a Gaussian mixture model, πk = p(Z = k) is the prior probability of selecting component k before observing an input; it can be modeled by that class’s fraction of the training population. In a variational autoencoder, a chosen latent prior such as a zero-mean, unit-covariance Gaussian regularizes latent vectors toward a compact region. section 6.9.2 section 14.7.7
Prior-guided height inference
If a neighboring population suggests adult female heights have mean 150 cm and standard deviation 9 cm, those values can initialize a normal-gamma prior for a new town. Combining that prior with observed samples yields posterior MAP estimates for the new town’s mean and standard deviation. section 13.8.2 section 13.7.2
