Bayesian statistics estimates unknown quantities by combining a likelihood from observed data with a prior belief. Bayes’ theorem turns them into a posterior distribution: p(θ|X) is proportional to p(X|θ)p(θ). The prior matters most with little data; as data grows, the likelihood increasingly dominates. section 6.6.1 section 13.11
Choose a likelihood family for the data, then express its parameters as uncertain variables. A Gaussian likelihood, for example, may use a Gaussian prior for an unknown mean, a gamma prior for unknown precision, or a normal-gamma prior when both mean and precision are unknown. Conjugate priors keep the posterior in a tractable distribution family. section 13.3 section 13.6.1 section 13.7.2
Bayesian parameter estimation distinguishes these quantities:
- Likelihood p(X|θ): how probable the observed training data is under parameters θ; for independent instances, it is the product of per-instance probabilities. section 6.6.1
- Prior p(θ): beliefs or constraints on parameters before using the data, such as favoring smaller-magnitude parameter vectors. section 6.6.1
- Posterior p(θ|X): the updated uncertainty over parameters after combining prior and likelihood. section 13.11
- MAP estimate: the parameter value at the maximum of the posterior; unlike MLE, it incorporates prior information. section 13.11
Estimating heights from a small sample
Model adult female heights as Gaussian when only five samples are available. A normal-gamma prior informed by a neighboring population supplies beliefs about mean and variation; updating it with the samples yields posterior MAP estimates for mean and standard deviation, then a fitted distribution for range probabilities. section 13.8 section 13.8.2
The same update supports unsupervised classification with Gaussian mixture models. After fitting mixture parameters, compute p(Z=k|X=x) for each cluster and assign an input to the cluster with the largest posterior probability; variance controls how tolerant each cluster is of distance from its mean. section 6.9.3
Listing 13.5 PyTorch- Computing posterior probability using Bayesian inference
prior_dist = NormalGamma(150, 100, 10.5, 810) #1 posterior_dist = inference_unknown_mean_variance(X, prior_dist) #2 map_mu, map_precision = posterior_dist.mode #3 map_std = math.sqrt(1 / map_precision) #4 map_dist = Normal(map_mu, map_std) #5
A short PyTorch workflow constructs a normal-gamma prior, computes its posterior from samples, extracts MAP mean and precision, and creates the fitted normal distribution. section 13.8.2
