Chapter 13: Fully Bayes model parameter estimation

Introduction

This introduction frames unsupervised modeling as fitting a probability density whose samples overlap the training-data cloud, enabling generation and probability evaluation of new examples.

• Unsupervised density-modeling objective: Explain that unsupervised modeling seeks a probability density function p() whose sampled point cloud largely overlaps the unknown data subspace, using the collected training samples X as a surrogate for that subspace.
  Why it matters: This objective defines what the chapter’s Bayesian parameter-estimation methods are trying to build.
• Generative model uses: Describe the two uses of the fitted probability model p(): sampling it to generate additional data examples and evaluating it on a new image to estimate that image’s probability under the cat-image model.
  Why it matters: A fitted density model is useful both for creating plausible examples and for scoring whether a new example matches the modeled data.

Key terms: unsupervised model, probability density function, sample cloud, subspace of interest, training data, X, generative modeling, generate data samples, evaluate p()

13.1 Fully Bayes estimation: An informal introduction

This section frames model estimation as choosing a likelihood family and fitting its parameters so its generated samples overlap the training data.

• Likelihood family and parameters: Describe the two aspects of likelihood estimation: selecting a likelihood-function family and estimating that family's parameters from training data. State that the family is usually preselected using knowledge of the problem.
  Why it matters: This separates the modeling assumption from the task of fitting the numerical parameter values.
• Likelihood fit as sample overlap: Explain that parameter values are chosen to make the overall likelihood of the training set high, expressed as the product of the individual sample likelihoods. Describe the section's geometric picture of a good fit as substantial overlap between the likelihood's sample cloud and the training data.
  Why it matters: The sample-overlap picture gives an intuitive meaning to fitting a probabilistic model to data.
• Gaussian fit geometry: Describe the geometric conditions for a well-fitted Gaussian likelihood: its mean lies where training points are highly concentrated, and its covariance gives an elliptical base that tightly contains as many training points as possible.
  Why it matters: This makes the abstract Gaussian parameters visible as location and spread of the fitted data cloud.

Key terms: likelihood function, function family, model parameters, training data, Gaussian, overall likelihood, p(X|θ), sample cloud, training data set, best fit

13.1.1 Parameter estimation and belief injection

This subsection contrasts data-only MLE with MAP estimation that injects prior beliefs, introducing fully Bayes estimation as Gaussian priors over unknown Gaussian parameters whose influence declines as data grows.

• MLE and MAP choice: Distinguish MLE from MAP parameter estimation: MLE maximizes the likelihood of the observed training data, whereas MAP maximizes posterior probability by combining likelihood with a prior. Explain why MLE is suitable with large training sets while MAP can use prior knowledge when data is insufficient.
  Why it matters: This distinction explains why Bayesian estimation can make useful parameter estimates when the data alone is limited.
• Prior belief injection: Describe how a prior probability density injects a belief about unknown parameters in the fully Bayes paradigm. For a Gaussian likelihood, explain that a Gaussian prior on the mean centered at zero expresses a belief that the mean is likely near zero, with its covariance representing confidence in that belief.
  Why it matters: A prior turns informal assumptions about a system into a mathematical influence on parameter estimates.
• Prior and data influence: Explain that with a low volume of training data, the prior can dominate the fully Bayesian estimate, but as sufficient data is incorporated, the prior's effect fades and the solution approaches the MLE estimate. Distinguish this parameter-distribution approach from the earlier MAP regularization example that used a prior proportional to exp(-||θ||²) to favor small parameter magnitudes.
  Why it matters: This explains when prior assumptions materially shape a model and when observed evidence takes over.

Key terms: maximum likelihood estimation, MLE, maximum a posteriori estimation, MAP, likelihood, posterior probability, fully Bayes parameter estimation, Gaussian likelihood, Gaussian prior, parameter space

13.2 MLE for Gaussian parameter values recap)

This section recaps maximum likelihood estimation of Gaussian parameters from training data and its use in evaluating an unknown point's probability.

• Gaussian MLE objective: Explain that Gaussian maximum likelihood estimation chooses the mean and variance parameters that maximize the likelihood, or log-likelihood, of the observed training dataset.
  Why it matters: This establishes the non-Bayesian parameter estimate that the chapter later contrasts with fully Bayesian estimation.
• Sample mean and variance estimates: State that the closed-form MLE estimates for a univariate Gaussian are the mean and variance of the training samples.
  Why it matters: These estimates provide the concrete MLE baseline for later Bayesian estimates of Gaussian parameters.
• Gaussian density evaluation: Describe how the fitted univariate or multivariate Gaussian MLE parameters are used to evaluate p(x), the probability density for an unknown data point under the distribution represented by the training dataset.
  Why it matters: A fitted distribution becomes useful when it can score how compatible a new point is with the training-data distribution.

Key terms: maximum likelihood estimation, MLE, likelihood, log-likelihood, Gaussian, closed-form solution, MLE mean, MLE variance, training data samples, p(x)

13.3 Fully Bayes parameter estimation: Gaussian, unknown mean, known precision

This section derives fully Bayesian estimation of an unknown Gaussian mean with known variance by combining a Gaussian prior with the data likelihood into a Gaussian posterior.

• Bayesian mean estimation: Explain how fully Bayesian estimation of an unknown Gaussian mean maximizes the posterior rather than the likelihood alone. State that the posterior combines a likelihood from the observed training data with a prior that encodes a belief about the mean.
  Why it matters: This is the basic way Bayesian estimation incorporates both evidence from data and prior knowledge.
• Gaussian posterior for mean: Describe the model in which data are Gaussian with known variance and unknown mean, while the unknown mean has a Gaussian prior with mean μ0 and variance σ0. Explain that multiplying the Gaussian likelihood by this Gaussian prior produces a Gaussian posterior p(μ|X) with parameters μn and σn after n samples.
  Why it matters: The Gaussian-prior/Gaussian-likelihood case gives a tractable example of Bayesian parameter updating.
• Roles of mean variables: Distinguish the data-distribution parameters μ and σ from the prior parameters μ0 and σ0 and posterior parameters μn and σn. State specifically that σn measures uncertainty in the estimate of μ, not the known variance σ of the data distribution.
  Why it matters: Keeping these similarly named quantities separate prevents confusing uncertainty about a parameter with variability in the data.
• Data-prior weighting behavior: Explain how the posterior mean μn combines the MLE sample mean x̄ with the prior guess μ0, and trace how their influence changes with sample size. State that little or no data leaves the estimate at the prior, abundant data makes the Bayesian estimate approach MLE, and posterior uncertainty σn approaches zero as data become very large.
  Why it matters: This behavior explains when prior beliefs meaningfully affect an estimate and when evidence from data overwhelms them.
• MAP estimate and inference: State that the posterior p(μ|X) is maximized at μ = μn, so the MAP estimate is μMAP = μn. Describe how the resulting fitted data distribution p(x) = N(x; μn, σ) can be used to infer the probability that an arbitrary instance belongs to the training-data class.
  Why it matters: The posterior is turned into a concrete parameter estimate and then into a distribution for evaluating new instances.

Key terms: fully Bayes parameter estimation, posterior probability, likelihood, prior probability, MAP estimate, Gaussian, known variance, unknown mean, Gaussian prior, Gaussian posterior

13.4 Small and large volumes of training data, and strong and weak priors

This section explains how posterior Gaussian parameter estimates shift between prior belief and data as training-set size and prior strength vary.

• Training-data influence on posterior: Explain how the number of training examples n changes the balance between the prior and likelihood in the posterior estimate of a Gaussian mean. State what happens with no or little data versus very large amounts of data, including the connection to the sample-mean MLE estimate.
  Why it matters: This explains when a Bayesian estimate is mainly expressing prior belief and when it is mainly determined by observed training data.
• Strong and weak priors: Describe how the prior variance σ0² determines whether a prior is strong or weak and how readily data can overcome it. Explain why a smaller prior variance requires more training examples before the posterior mean approaches the MLE mean.
  Why it matters: Prior strength determines how much confidence a Bayesian model assigns to beliefs supplied before seeing the data.

Key terms: posterior, prior, likelihood, training data, sample mean, MLE estimate, prior variance, σ0, strong prior, weak prior

13.5 Conjugate priors

This section defines conjugate priors and shows how they keep Bayesian posterior distributions in tractable distribution families for Gaussian likelihoods.

• Conjugate prior definition: Explain that a conjugate prior is a prior chosen for a particular likelihood family such that the posterior belongs to the same distribution family as the prior. Describe why this is useful: multiplying likelihood and prior then yields a simple, known posterior family rather than an unfamiliar or intractable one.
  Why it matters: Conjugate priors make fully Bayesian parameter estimation analytically manageable.
• Gaussian conjugate-prior families: State the conjugate-prior examples for a Gaussian likelihood: a Gaussian prior for an unknown mean with known variance gives a Gaussian posterior, a gamma prior for precision gives a gamma posterior, and the multivariate precision counterpart uses a Wishart prior.
  Why it matters: These distribution pairings identify the tractable prior choices used in the following Gaussian Bayesian-estimation cases.

Key terms: conjugate prior, likelihood, prior, posterior, Gaussian likelihood, Gaussian prior, Gaussian posterior, precision, gamma distribution, Wishart distribution

13.6 Fully Bayes parameter estimation: Gaussian, unknown precision, known mean

This section sets up fully Bayesian estimation of an unknown Gaussian precision with a known mean, using a gamma prior that yields a gamma posterior.

• Precision-based Gaussian model: Describe the fully Bayesian setup in which Gaussian data have a known constant mean μ while precision λ is unknown and treated as a random variable. State that precision is used instead of variance because λ=1/σ², and that the data likelihood is Gaussian conditioned on μ and λ.
  Why it matters: This setup changes uncertainty about the Gaussian spread into a probability distribution over precision.
• Gamma conjugate posterior: Explain why a gamma distribution is chosen as the prior for precision λ and what posterior results after observing Gaussian data. State that the Gaussian likelihood times the gamma prior produces another gamma distribution by conjugacy, with posterior parameters obtained by coefficient comparison.
  Why it matters: Conjugacy makes Bayesian updating for the unknown precision tractable and keeps the posterior in a familiar distribution family.
• Posterior-mode precision estimate: State that the parameter estimate is taken as the maximum, or mode, of the posterior gamma density. Describe finding that maximum by differentiating the posterior density with respect to λ and setting the derivative to zero.
  Why it matters: The posterior mode turns the full Bayesian posterior over precision into a single precision estimate when one is needed.

Key terms: precision, variance, Gaussian distribution, known mean, random variable, likelihood, gamma distribution, gamma prior, conjugate prior, posterior

13.6.1 Estimating the precision parameter

This subsection derives Bayesian estimation of a Gaussian's unknown precision with known mean, using a gamma prior and a MAP estimate.

• Gamma posterior for precision: Describe how a Gaussian likelihood with known mean and unknown precision is combined with a gamma prior on precision to produce a gamma posterior for λ. State that this preservation of distribution family occurs because the gamma is conjugate to the Gaussian precision likelihood.
  Why it matters: Conjugate priors make Bayesian parameter estimation analytically tractable rather than requiring a numerical posterior calculation.
• Prior-data influence balance: Explain how the amount of training data changes the relative influence of the gamma prior and the likelihood on the posterior precision estimate. State that the prior dominates with little data, whereas the data-based likelihood dominates as n becomes large.
  Why it matters: This explains the practical Bayesian tradeoff between injecting prior belief when evidence is scarce and letting abundant evidence drive estimation.
• MAP precision estimate: State that the MAP precision estimate is the value of λ that maximizes the gamma posterior, namely λMAP=(αn−1)/βn, equivalently 1/σMAP². Describe that the fitted data distribution is then a Gaussian with the known mean and σMAP, and distinguish its limiting behavior: it equals the MLE with abundant data and is determined entirely by the prior with no data.
  Why it matters: A MAP point estimate converts the full posterior over precision into parameters for a concrete Gaussian model.

Key terms: precision, lambda, Gaussian likelihood, gamma distribution, gamma prior, posterior, prior, likelihood, training data, n

13.7 Fully Bayes parameter estimation: Gaussian, unknown mean, unknown precision

This section introduces the normal-gamma distribution as the conjugate prior when both a Gaussian's mean and precision are unknown.

• Normal-gamma conjugate prior: Explain why a normal-gamma distribution is used as the conjugate prior for a Gaussian model whose mean and precision are both unknown. Relate it to the Gaussian prior for an unknown mean with known variance and the gamma prior for an unknown precision with known mean.
  Why it matters: This identifies the prior distribution needed to estimate both central tendency and uncertainty in the same Gaussian model.

Key terms: normal-gamma distribution, conjugate prior, Gaussian distribution, normal distribution, mean, precision

13.7.1 Normal-gamma distribution

This subsection introduces the normal-gamma distribution as a four-parameter joint distribution for the random variables μ and λ, remembered as a normal distribution times a gamma distribution.

• Normal-gamma distribution structure: Describe the normal-gamma distribution as a probability distribution jointly defined over two random variables, μ and λ, with density specified by the four parameters μ′, λ′, α′, and β′. Explain the mnemonic that it can be viewed as the product of a normal distribution and a gamma distribution.
  Why it matters: This distribution provides the probabilistic form used when estimating a Gaussian mean and precision together.

Key terms: normal-gamma distribution, μ, λ, μ′, λ′, α′

13.7.2 Estimating the mean and precision parameters

This subsection derives Bayesian estimates for a Gaussian distribution’s unknown mean and precision by combining data likelihood with a normal-gamma prior.

• Normal-gamma model construction: Describe the probabilistic model for Gaussian data with both mean and precision unknown. Explain that the data likelihood is Gaussian, the mean has a Gaussian prior conditioned on precision, the precision has a gamma prior, and together these priors form a normal-gamma distribution.
  Why it matters: This model lets prior beliefs and observed data jointly determine estimates of both central tendency and uncertainty.
• Posterior parameter estimation: Explain how Bayes’ theorem combines the Gaussian likelihood and normal-gamma prior into a normal-gamma posterior through conjugacy. Describe that posterior parameters are found by substitution and coefficient comparison, and that the parameter estimate is obtained by maximizing this posterior density before forming the final data density.
  Why it matters: Conjugacy makes joint Bayesian estimation of mean and precision analytically tractable rather than requiring a separate numerical posterior calculation.

Key terms: Gaussian, precision, lambda, variance, Gaussian prior, gamma prior, posterior, Bayes' theorem, conjugate prior, normal-gamma posterior

13.8 Example: Fully Bayesian inferencing

This section sets up a small-sample height-modeling example in which fully Bayesian inference will estimate a Gaussian distribution’s mean and standard deviation.

• Small-sample height modeling: Describe the Statsville example: a model is built from a very small sample of adult female heights to predict whether a resident is female based on height. Explain why the small sample motivates trying a Bayesian approach rather than abandoning the modeling task.
  Why it matters: This example provides the concrete small-data problem used to compare maximum likelihood and Bayesian inference.
• Gaussian height parameter target: State that the section assumes adult female heights follow a Gaussian distribution and that the parameters to estimate are its mean μ and standard deviation σ. Distinguish these unknown real-world distribution parameters from the values chosen to generate the illustrative sample.
  Why it matters: Fully Bayesian inference here is about estimating uncertainty in the parameters of an assumed Gaussian height distribution.
• Illustrative five-point dataset: Describe the example data setup: five height values are sampled from a Gaussian with μ = 152 cm and σ = 8 cm to create the data matrix X. State that these values are supplied only so the example can be constructed, although the true mean and standard deviation would ordinarily be unknown.
  Why it matters: The deliberately tiny synthetic dataset makes the later parameter-estimation comparison concrete.

Key terms: Statsville, fully Bayesian inferencing, training data, Gaussian distribution, μ, σ, mean, standard deviation, data matrix X, five points

13.8.1 Maximum likelihood estimation

This subsection shows that maximum likelihood estimation fits a normal model using the sample mean and standard deviation, but can be unreliable with little data.

• Gaussian MLE fitting: Describe how MLE fits a normal distribution to a data set by computing the data's mean and standard deviation and using them as the model parameters.
  Why it matters: This makes clear what an MLE-based Gaussian model represents before the section contrasts it with Bayesian inference.
• Low-data MLE unreliability: Explain why the MLE estimates in this example are unreliable: the small number of data points produces a fitted mean of 149.68 and standard deviation of 11.52, substantially different from the true values 152 and 8.
  Why it matters: It motivates using a Bayesian approach when limited data make estimates based only on observations unstable.

Key terms: MLE, mean, standard deviation, normal distribution, maximum likelihood estimates, true mean, true standard deviation, low-data scenarios

13.8.2 Bayesian inference

This subsection applies fully Bayesian inference to estimate adult-height distribution parameters using a prior informed by a neighboring town and compares the resulting MAP estimates with MLE.

• Survey-informed normal-gamma prior: Explain how information from an earlier Neighborville survey is used to set a prior for Statsville adult female heights when both the Gaussian mean and variance are unknown. State that the prior encodes beliefs about parameter values and is modeled as a normal-gamma distribution.
  Why it matters: A meaningful prior lets Bayesian estimation incorporate relevant knowledge that is not contained in the current sample.
• Posterior parameter estimation: Describe that the selected normal-gamma prior is combined with the Statsville observations to obtain a normal-gamma posterior, whose parameters are computed using the equations from section 13.7. State the specific chosen prior settings: μ0=150, λ0=100, α0=10.5, and β0=810.
  Why it matters: The posterior is the Bayesian result that updates initial beliefs using the observed training data.
• MAP and MLE comparison: State the Bayesian MAP estimates for the height distribution and compare them with the MLE estimates given in the example. Explain that the Bayesian estimates, μ=149.98 and σ=9.56, are closer to the stated true values μ=152 and σ=9 than the MLE values μ=149.68 and σ=11.52.
  Why it matters: The numerical comparison illustrates how a good prior can improve parameter estimates over MLE for this data example.
• Range probability after estimation: Describe that, once the distribution parameters have been estimated, the model can be used to calculate the probability that a sample falls within a specified range.
  Why it matters: Parameter estimation is useful because it supports probability questions about future or sampled values.

Key terms: prior distribution, normal-gamma distribution, unknown mean, unknown variance, parameter values, posterior distribution, p(θ), p(θ|X), μ0, λ0

13.9 Fully Bayes parameter estimation: Multivariate Gaussian, unknown mean, known precision

This section extends fully Bayesian mean estimation to multivariate Gaussian data with known precision, deriving a Gaussian posterior and its MAP mean estimate.

• Multivariate Gaussian Bayesian setup: Describe the multivariate Gaussian model for data with an unknown mean and known precision matrix, including the Gaussian prior placed on the mean. State that the precision matrix Λ is the inverse of the covariance matrix Σ.
  Why it matters: This setup lets Bayesian estimation represent uncertainty about a vector-valued mean rather than treating each feature independently.
• Gaussian posterior mean update: Explain why multiplying the multivariate Gaussian likelihood by a Gaussian prior yields a Gaussian posterior for the mean. Describe the MAP estimate as the posterior mean μ_n, formed by combining the prior mean μ_0 with the sample mean using their respective precision contributions.
  Why it matters: The posterior update shows concretely how Bayesian estimation combines observed data with an initial belief about a parameter.
• Data and prior limiting behavior: Trace how the MAP mean estimate changes with the amount of training data: with much data it approaches the MLE sample mean, while with little data it approaches the prior mean μ_0.
  Why it matters: This limiting behavior makes clear when observations override prior beliefs and when prior beliefs have substantial influence.

Key terms: multivariate Gaussian, mean parameter, precision matrix, covariance matrix, Gaussian prior, likelihood, posterior probability density, Gaussian posterior, MAP estimate, posterior mean

13.10 Fully Bayes parameter estimation: Multivariate, unknown precision, known mean

This section introduces multivariate fully Bayesian precision estimation with a known mean by connecting it to the earlier univariate case and identifying the Wishart distribution as the relevant distribution.

• Wishart distribution role: State that the multivariate version of fully Bayesian parameter estimation with unknown precision and known mean uses the Wishart distribution, whereas the earlier univariate case used the gamma distribution.
  Why it matters: This identifies the distributional tool needed to extend Bayesian precision estimation from one variable to multiple variables.

Key terms: Wishart distribution, gamma distribution, multivariate case, univariate case, precision

13.10.1 Wishart distribution

This subsection introduces the Wishart distribution as a prior distribution for an unknown multivariate Gaussian precision matrix.

• Wishart matrix domain: Explain why an unknown precision matrix requires a distribution over symmetric non-negative definite matrices rather than an ordinary distribution over arbitrary matrix entries. State that a symmetric d × d precision matrix has d(d + 1)/2 distinct elements, because entries on opposite sides of the diagonal are identical.
  Why it matters: A prior for a precision matrix must respect the structural constraints that make it a valid covariance or precision-related matrix.
• Wishart prior parameters: Describe the Wishart distribution as a distribution for a random d × d precision matrix Λ, parameterized by a scalar ν with ν > d − 1 and a d × d symmetric non-negative definite matrix W. State that it is the multivariate analogue of the gamma distribution and that its density uses determinants, a trace term, and the multivariate gamma function.
  Why it matters: The Wishart supplies a mathematically valid Bayesian prior for multivariate precision when a scalar gamma prior is no longer sufficient.
• Precision and covariance inversion: State that the precision matrix is denoted by Λ and is the inverse of the covariance matrix Σ, so Λ = Σ⁻¹ in this multivariate Gaussian setting. Describe the subsection's estimation case as known mean with unknown precision.
  Why it matters: Keeping precision and covariance distinct is necessary to follow which Gaussian parameter the Wishart prior is modeling.

Key terms: precision matrix, covariance matrix, symmetric matrix, non-negative definite matrix, Wishart ensemble, Wishart distribution, Wishart, ν, W, determinant

13.10.2 Estimating precision

This subsection derives the Wishart posterior and MAP estimate for a multivariate Gaussian precision matrix when the mean is known.

• Wishart posterior for precision: Describe how a multivariate Gaussian with known mean and unknown precision matrix is estimated in the fully Bayesian setting using a Wishart prior. State that the posterior over the precision matrix is also Wishart because the Wishart is conjugate to the Gaussian precision likelihood.
  Why it matters: Conjugacy makes Bayesian precision estimation analytically tractable instead of requiring numerical posterior approximation.
• Trace likelihood rewriting: Explain how the Gaussian likelihood exponent is rewritten with traces so that it can be combined with the Wishart prior. State the two used facts: a scalar quadratic form equals its trace, and the trace can cyclically reorder a matrix product.
  Why it matters: Writing the likelihood in trace form exposes the Wishart form and lets the posterior parameters be identified.
• Posterior parameter updates: Trace how the posterior Wishart parameters are obtained by comparing the combined prior and likelihood with a Wishart density. Describe that the training examples contribute through their deviations from the known mean, updating the posterior degrees of freedom and scale parameter.
  Why it matters: The updated Wishart parameters summarize how both prior belief and all observed data determine the uncertainty about precision.
• MAP precision estimate: State that the maximum of the posterior Wishart density provides a point estimate of the precision matrix, namely Lambda equals (nu_n minus d minus 1) times W_n, subject to the stated degrees-of-freedom condition.
  Why it matters: The posterior mode converts the full Bayesian posterior into a single usable precision-matrix estimate when a point estimate is needed.

Key terms: precision matrix, covariance matrix, Wishart distribution, Wishart conjugate prior, posterior, multivariate Gaussian, trace, quadratic form, Tr, likelihood exponent

13.11 Summary

This summary recaps fully Bayesian parameter estimation as updating prior beliefs with data likelihoods to obtain posterior distributions for Gaussian model parameters and predictions.