Chapter 6: Bayesian tools for machine learning

Introduction

This chapter introduction contrasts Bayesian and frequentist views and frames unsupervised learning as fitting probability distributions to patterns in unlabeled data.

• Bayesian and frequentist perspectives: Distinguish the Bayesian approach, which models uncertainty together with prevailing beliefs and knowledge about a system, from the frequentist view, which measures probability as an event’s observed long-run frequency.
  Why it matters: This distinction explains why Bayesian probability tools are especially relevant to the kind of machine learning developed in this chapter.
• Unsupervised distribution modeling: Describe unsupervised machine learning as building a model from unlabeled training data by finding a probability distribution whose sample-point cloud matches the pattern or shape of the training-data points, potentially after a transformation.
  Why it matters: This is the chapter’s geometric picture of how unlabeled data can be modeled without manually supplied desired outputs.
• Distribution-model applications: Explain that a distribution fitted to unlabeled data can generate faux inputs by sampling and can classify an arbitrary input by determining which data cluster it falls into.
  Why it matters: These uses show why identifying the probability structure and clusters in unlabeled data is useful.

Key terms: Bayesian approach, frequentist paradigm, probability, unsupervised machine learning, unlabeled training data, probability distribution, sample point cloud, transformation, sampling, faux input

6.1 Conditional probability and Bayes’ theorem

This section sets up a height-and-weight example using a joint probability table to refresh joint and marginal probability before conditional probability and Bayes' theorem.

• Joint probability table setup: Describe the example of two random variables, adult residents' quantized weight W and height H, and state that their joint probabilities are represented in Table 6.1.
  Why it matters: This shared example provides the probability data used to introduce the Bayesian tools that follow.

Key terms: joint probability, marginal probability, random variables, weight W, height H, Table 6.1

6.1.1 Joint and marginal probability revisited

This subsection revisits how a joint probability table represents paired events, yields marginal probabilities by summation, and tests whether variables are independent.

• Joint probability interpretation: Explain joint probability as the probability that a specific combination of values for two variables occurs together, and identify a table cell such as p(E2,F3) as one such probability.
  Why it matters: Joint probabilities are the starting point for reasoning about relationships between random variables.
• Marginal probability summation: Describe how to obtain a variable's marginal probability by summing away all possible values of the other variable, and state that the full set of marginal probabilities sums to 1.
  Why it matters: Marginals reduce a joint distribution to the probabilities for one variable and are needed to assess independence.
• Joint distribution sum rule: State that the joint probabilities across all possible combinations of the variables sum to 1, because one of the exhaustive combinations must occur.
  Why it matters: This sum rule checks that a joint probability distribution accounts for all possible outcomes.
• Independence product criterion: Distinguish dependent from independent variables using the criterion that p(Ei,Fj) equals p(Ei) times p(Fj) for every valid value pair if and only if the variables are independent; explain that the weight-height example fails this test.
  Why it matters: The product test reveals whether knowing one variable changes the probability structure of the other.

Key terms: joint probability, joint probability distribution, marginal probability, summing away, joint probabilities, sum rule, exhaustive combinations, dependent variables, independent variables, marginal probabilities

6.1.2 Conditional probability

This section introduces conditional probability as the probability of an event within the restricted population satisfying a given condition.

• Conditional probability notation: Describe the notation p(W = E3|H = F2) and read it as the probability that W equals E3 given, or subject to the condition, that H equals F2.
  Why it matters: Conditional notation makes explicit which information is assumed when stating a probability.
• Restricted conditional population: Explain that conditioning on H = F2 restricts the universe to subjects in the H = F2 row, so the relevant population is 48,000 rather than the full Statsville population of 100,000.
  Why it matters: Conditional probabilities must be computed relative to the population that meets the stated condition, not the entire population.

Key terms: conditional probability, p(W = E3|H = F2), given, subject to the condition, universe, given condition, H = F2, population size, frequentist definition

6.1.3 Bayes’ theorem

This subsection states Bayes’ theorem as conditional probability computed from a joint probability divided by the marginal probability of the condition, and shows its consequence for independent variables.

• Bayes theorem probability ratio: Explain Bayes’ theorem for variables X and Y: p(X=x|Y=y) is the joint probability p(X=x,Y=y) divided by the marginal probability p(Y=y). Describe that the denominator is found by summing or integrating the joint probabilities over the possible values of X.
  Why it matters: Bayes’ theorem provides the probability relationship used throughout Bayesian parameter estimation and inference.
• Independent-variable conditional probability: State the independence consequence of Bayes’ theorem: when X and Y are independent, their joint probability factors into their marginals and p(X=x|Y=y) equals p(X=x). Explain that observing Y then provides no information about X.
  Why it matters: This gives an intuitive test for when conditioning should and should not change a probability.

Key terms: Bayes’ theorem, conditional probability, joint probability, marginal probability, independent random variables

6.2 Entropy

This section defines entropy as a measure of uncertainty in a probability distribution and relates low entropy to greater potential for variable-bit-rate compression.

• Entropy as distributional uncertainty: Explain entropy as the overall uncertainty associated with a probability distribution. Describe why entropy is low when one outcome is overwhelmingly likely and high when multiple outcomes have comparable probabilities.
  Why it matters: Entropy gives machine learning a numerical way to describe how predictable or uncertain data and outcomes are.
• Entropy and data compression: Describe how entropy predicts the usefulness of variable bit-rate coding: frequent, high-probability symbols can receive short codes, while rare symbols receive longer codes. Explain why a distribution dominated by a few symbols can be compressed more effectively than one with equally frequent symbols.
  Why it matters: This connection makes entropy concrete: predictable data require fewer bits on average to encode.
• Discrete and continuous entropy formulas: State the discrete entropy definition as the negative sum of each outcome probability times its natural logarithm, −Σ p(x) log p(x). Describe that entropy also has continuous univariate and multidimensional forms, replacing the discrete sum with an integral over the variable's domain.
  Why it matters: The formulas let you calculate entropy from a distribution rather than relying only on an informal idea of uncertainty.

Key terms: entropy, uncertainty, probability distribution, random variable, variable bit-rate coding, prefix coding, Huffman encoding, fixed bit-rate coding, compression, discrete entropy

6.2.1 Geometrical intuition for entropy

This subsection gives a geometric intuition for entropy by contrasting flat, diffuse distributions with peaked, concentrated ones.

• Entropy as distribution lopsidedness: Explain entropy as a measure of how lopsided a probability density function is, relating a roughly uniform distribution to high entropy and a distribution with a few disproportionately likely inputs to low entropy.
  Why it matters: This lets you recognize uncertainty directly from the shape of a distribution.
• Density and sample-cloud geometry: Describe how the shape of a PDF corresponds to the distribution of sample points: a flat PDF produces a diffuse cloud without highly concentrated regions, whereas PDF peaks correspond to dense regions of points.
  Why it matters: Connecting density curves to point clouds provides a visual way to interpret probability distributions.
• Probability-sum constraint: State that probabilities must sum to one, so when a few possible inputs have high probabilities, the remaining inputs must have lower probabilities; all probabilities cannot simultaneously be high or low.
  Why it matters: This constraint explains why a distribution with pronounced peaks must also have relatively low-probability regions.

Key terms: entropy, probability density function, PDF, uncertainty, sample point cloud, density function, concentration of points, probabilities, sum of all probabilities, 1

6.2.2 Entropy of Gaussians

This subsection derives Gaussian entropy, showing that broader Gaussian distributions have greater uncertainty, and states the multivariate result.

• Gaussian spread and entropy: Explain why a wider, less peaked univariate Gaussian has higher entropy than a narrow Gaussian. State that its spread parameter, σ, makes Gaussian entropy increase as the distribution becomes fatter and closer to uniform.
  Why it matters: This connects entropy to an intuitive property of a distribution: how dispersed and uncertain its outcomes are.
• Univariate Gaussian entropy derivation: Trace the derivation of univariate Gaussian entropy from the definition of entropy and the Gaussian PDF. State that normalization makes the PDF integral equal one and that the expected squared deviation term is the variance, yielding the closed-form entropy expression.
  Why it matters: The derivation shows how entropy formulas arise from probability normalization and expectation rather than being arbitrary formulas.
• Multivariate Gaussian entropy formula: State the entropy result for a multivariate Gaussian and describe that it generalizes the univariate result to multiple dimensions through its covariance structure.
  Why it matters: Multivariate Gaussian entropy is needed whenever uncertainty is represented jointly across several features.

Key terms: Gaussian, univariate Gaussian, variance, σ, entropy, uniform distribution, probability density function, PDF, probability sum rule, expected value

6.3 Cross-entropy

This section defines cross-entropy as a loss that measures how dissimilar a classifier's predicted probability distribution is from the one-hot ground-truth distribution.

• Cross-entropy classification loss: Explain how cross-entropy quantifies the loss between a ground-truth class distribution and a network's predicted class probabilities in supervised classification. State that training adjusts model parameters to reduce this loss.
  Why it matters: Cross-entropy supplies a principled numerical signal for training a classifier to assign high probability to the correct class.
• One-hot ground-truth weighting: Describe how a ground-truth class is represented by a one-hot probability vector and how this makes cross-entropy focus on the probability assigned to the correct class. Explain that the zero entries for incorrect classes contribute nothing directly to the sum.
  Why it matters: One-hot targets explain why multiclass cross-entropy penalizes a model specifically for failing to assign probability to the correct class.
• Logarithmic penalty behavior: Explain why cross-entropy is small when the predicted probability of a high-probability ground-truth outcome is near 1 and becomes very large when that predicted probability is near 0. Relate this behavior to the negative logarithm and to probabilities in each distribution summing to 1.
  Why it matters: The logarithmic penalty makes confidently assigning almost zero probability to the correct class especially costly.
• General cross-entropy definition: State the general definition of cross-entropy for two random variables defined on the same domain: the negative sum of one distribution times the logarithm of the other distribution. Explain that it reduces to entropy when the two distributions are the same.
  Why it matters: The general definition lets the same idea apply beyond one-hot multiclass classification.

Key terms: cross-entropy, cross-entropy loss, ground truth, PDF, predicted distribution, one-hot vector, pgt, ppred, probability sum rule, negative logarithm

6.4 KL divergence

This section defines KL divergence as cross-entropy with the target distribution’s entropy offset removed, and establishes its key properties for comparing probability distributions and training classifiers.

• KL divergence definition: Explain KL divergence as the pure dissimilarity between two probability distributions, obtained by subtracting the entropy of the first distribution from their cross-entropy. State that it compares distributions using probability-weighted log ratios and can be defined for discrete, continuous univariate, and continuous multivariate random variables.
  Why it matters: KL divergence isolates distribution mismatch from the unavoidable entropy already present in the reference distribution.
• KL divergence properties: Describe the principal properties of KL divergence: it is zero for identical distributions, positive for non-identical distributions, and not symmetric when its two arguments are exchanged. Explain why minimizing cross-entropy between a fixed ground-truth distribution and a prediction is equivalent to minimizing their KL divergence.
  Why it matters: These properties explain why cross-entropy can train a classifier even though cross-entropy itself is not zero when prediction equals ground truth.

Key terms: Kullback–Leibler divergence, KL divergence, KLD, cross-entropy, entropy, probability distribution, identical random variables, non-identical probability distributions, ground-truth PDF, machine prediction

6.4.1 KLD between Gaussians

This subsection gives the KL divergence formulas for univariate and multivariate Gaussian distributions and illustrates how differences in their means or spreads affect the divergence.

• Gaussian KLD formulas: Describe the KLD between two Gaussian random variables in both the univariate and d-dimensional cases. State that the multivariate expression compares their means and covariance matrices and uses matrix trace and determinant operations.
  Why it matters: Gaussian KLD provides a concrete way to measure how different two commonly used probability models are.
• Mean and spread effects: Explain how Gaussian KLD changes when distributions differ in standard deviation or mean. Describe the examples in which divergence is smaller when the compared distribution has a spread closer to the reference distribution, or a mean closer to it.
  Why it matters: This gives an intuitive interpretation of why two distributions with more similar location or spread are judged closer by KLD.
• PyTorch KLD computation: State that the subsection provides PyTorch code for computing KLD between Gaussian distributions and identify Listing 6.3 as the code example.
  Why it matters: The code connects the Gaussian KLD equations to an implementable calculation.

Key terms: KLD, Gaussian random variables, univariate Gaussian, multivariate Gaussian, trace, determinant, mean, standard deviation, Gaussian distribution, PyTorch

6.5 Conditional entropy

This section defines conditional entropy as the uncertainty remaining in one random variable after conditioning on another and shows how to compute it as a probability-weighted average.

• Conditional entropy meaning: Explain conditional entropy as the entropy, or uncertainty, of one random variable when the value of another variable is known. Describe how conditioning restricts attention to the subset of outcomes consistent with that known value.
  Why it matters: Conditional entropy quantifies the uncertainty that remains after information about another variable is available.
• Conditional entropy computation: Describe how to compute H(W|H) by first finding H(W|H = Fi) for each possible value Fi of H, then taking their probability-weighted average. State the general form H(X|Y) = sum over y of P(Y = y) times H(X|Y = y).
  Why it matters: The weighted-average calculation turns conditional uncertainty for individual conditions into an overall conditional entropy.

Key terms: conditional entropy, entropy, conditional probability, random variables, H(W|H = Fi), H(W|H), expected value, probability-weighted average

6.5.1 Chain rule of conditional entropy

6.6 Model parameter estimation

This section introduces model parameter estimation as the task of determining a chosen model family's unknown parameters from unlabeled training samples.

• Model parameter estimation task: Explain why selecting a model family is insufficient by itself and why its parameter values must be estimated before the model can be used. Describe the setup as unlabeled sampled inputs used as training data.
  Why it matters: A model becomes usable only when its general form is paired with concrete parameter values.
• Generic parameter notation: State that the section denotes a model's parameters generically by theta, and relate theta to a Gaussian example in which theta consists of the mean and covariance.
  Why it matters: Generic notation lets the same estimation ideas apply before committing to a particular model architecture.

Key terms: training data, unlabeled training data, model family, model parameters, parameter estimation, θ, Gaussian, mean, covariance, Σ

6.6.1 Likelihood, evidence, and posterior and prior probabilities

This section defines the Bayesian probability terms used in parameter estimation, introduces MLE and MAP as optimization paradigms, and explains why likelihoods are commonly optimized in log form.

• Bayesian parameter probability terms: Explain likelihood, prior, posterior, and evidence in Bayes’ theorem for estimating a model’s unknown parameters from training data. State how these quantities relate as posterior probability being proportional to likelihood times prior, with evidence as the data probability term.
  Why it matters: These terms specify exactly which probability an estimation method is trying to maximize.
• Independent-data likelihood construction: Describe how the likelihood for a training set is formed from the selected distribution family’s probability density function when individual data instances are independent. Explain that the likelihood is the product of the per-instance probabilities or densities given the parameters.
  Why it matters: It connects a chosen probabilistic model, such as a Gaussian, to the objective used to fit its parameters.
• Prior preference and MAP rationale: Explain that a prior can encode physical constraints or a preference for smaller-magnitude parameter vectors without referring to the input data. Relate this preference to Occam’s razor and distinguish the MLE objective of maximizing likelihood from the MAP objective of maximizing posterior probability.
  Why it matters: Priors provide a principled way to favor simpler parameter settings rather than fitting data alone.
• Log-likelihood optimization: Explain why maximizing a likelihood is equivalent to maximizing its logarithm and why log-likelihood simplifies likelihoods built from exponential probability functions. State that logarithms turn products of exponential terms into sums of their exponents.
  Why it matters: The log-likelihood trick makes common estimation objectives, especially Gaussian ones, much easier to manipulate and optimize.

Key terms: likelihood, prior probability, posterior probability, evidence, Bayes’ theorem, p(θ), probability density function, PDF, independent data instances, Gaussian

6.6.2 Maximum likelihood parameter estimation (MLE)

This section defines maximum likelihood estimation as choosing model parameters that maximize the likelihood of the training data and outlines how that maximization is solved.

• Maximum likelihood objective: Explain that maximum likelihood estimation (MLE) chooses the parameter values θ that maximize the joint likelihood p(X|θ) of the training data instances.
  Why it matters: MLE provides a principled way to fit a probability-distribution model to observed training data.
• Distribution overlap intuition: Describe the geometric intuition for MLE: estimate a model distribution whose samples would form a point cloud that largely overlaps the training-data points.
  Why it matters: This picture connects an abstract likelihood objective to the practical idea of fitting a distribution to data.
• Likelihood maximization methods: State that practitioners often maximize log-likelihood instead of likelihood, and distinguish models with analytical closed-form MLE solutions from models requiring iterative maximization.
  Why it matters: The form of the model determines whether MLE can be solved directly or must be optimized iteratively.

Key terms: maximum likelihood estimation, MLE, joint likelihood, p(X|θ), parameters, θ, sample point cloud, training data, model probability distribution, likelihood

6.6.3 Maximum a posteriori (MAP) parameter estimation and regularization

This section defines MAP estimation as maximizing the posterior over parameters, interprets the prior as a regularizer added to MLE, and explains how data progressively outweigh prior beliefs.

• MAP posterior objective: Explain that maximum a posteriori (MAP) estimation chooses the parameter value θ that maximizes the posterior probability p(θ|X). State that, because the evidence denominator is independent of θ, this is equivalent to maximizing the product of the likelihood p(X|θ) and prior p(θ).
  Why it matters: MAP provides a way to estimate model parameters using both observed training data and prior beliefs.
• Prior as regularization: Describe why MAP is equivalent to MLE with regularization: the prior probability term is added on top of the likelihood objective. Explain that this extra term incentivizes lower-magnitude parameter values while the likelihood still favors parameters that explain the training data.
  Why it matters: This connection explains regularization as incorporating a preference among parameter solutions rather than merely adding an arbitrary optimization term.
• Data versus prior influence: Trace how the relative influence of the prior and likelihood changes as training data arrives. State that without training data the estimate comes from the prior, whereas with increasing data the fixed prior term matters less and the posterior becomes more dominated by the likelihood.
  Why it matters: This explains when prior beliefs strongly shape a MAP estimate and when observed data become the main driver.

Key terms: maximum a posteriori, MAP, posterior probability, likelihood, prior probability, evidence, regularization, MLE, training data

6.7 Latent variables and evidence maximization

This section introduces latent variables as hidden modeling aids and explains estimating their distributions by maximizing the evidence of observed training data.

• Latent-variable modeling: Explain what a latent or hidden variable is and how it can make a complicated observed-data distribution easier to model as a combination of simpler distributions. Use the height-and-weight example to describe a discrete class variable Z for male and female data.
  Why it matters: Latent variables let a model represent meaningful hidden structure even when the training data has no explicit labels.
• Evidence maximization principle: Describe evidence maximization for latent-variable models: choose p(X|Z) and p(Z) so that the observed training data have the greatest probability under the resulting model. Explain the geometric interpretation that the model distribution's probability mass should align with the region occupied by the training-data cloud.
  Why it matters: This gives a criterion for fitting hidden-variable distributions from observed data even though the hidden values themselves are not supplied.

Key terms: latent variables, hidden variables, hidden parameters, observed variables, Z, X, evidence maximization, training data, p(X|Z), p(Z)

6.8 Maximum likelihood parameter estimation for Gaussians

This section derives maximum-likelihood estimates for a Gaussian’s mean and variance from training samples and uses the fitted distribution to evaluate heights.

• Gaussian MLE fitting: Explain how maximum likelihood estimation fits a Gaussian with unknown mean and variance to one-dimensional training samples. State that the fitted Gaussian is the one whose sample cloud maximally matches the distribution of the training data.
  Why it matters: This provides a concrete example of turning observed data into the parameters of a probabilistic model.
• Sample mean and variance estimates: State that maximizing the Gaussian log-likelihood yields closed-form estimates: the MLE mean is the sample mean and the MLE variance is the sample variance of the training data. Describe that the log-likelihood trick is used to make the maximization and differentiation manageable.
  Why it matters: These closed-form estimates let a practitioner fit a Gaussian directly from data rather than search blindly for its parameters.
• Gaussian probability evaluation: Describe how the estimated mean and standard deviation define a Gaussian distribution that can be used to compute the probability for a height or for heights in a specified range [a, b]. State that the one-dimensional derivation is presented as an example but the results extend to higher dimensions.
  Why it matters: A fitted distribution is useful because it can evaluate new observations and ranges, not merely summarize the training set.

Key terms: maximum likelihood estimation, MLE, Gaussian distribution, training data, mean, variance, log-likelihood trick, sample mean, sample variance, closed-form expression

6.8.1 Python PyTorch code for maximum likelihood estimation

This subsection frames Gaussian maximum likelihood estimation in PyTorch as learning unknown distribution parameters from height training data to create a predictive model.

• Gaussian parameter estimation: Explain how maximum likelihood estimation uses a dataset of observed values to estimate an unknown Gaussian's mean μ and standard deviation σ when the Gaussian family is assumed known.
  Why it matters: Estimating these parameters turns raw training examples into a distribution that can make probability-based predictions.
• Predictive Gaussian model: Describe how estimated Gaussian parameters define a complete estimated distribution that can predict an expected height or the probability that a height falls within a specified range, such as 160 to 170 cm.
  Why it matters: The value of parameter estimation is that the fitted distribution answers questions about future or unobserved cases.

Key terms: maximum likelihood estimation, MLE, Gaussian distribution, mean, standard deviation, training data, estimated distribution, model, expected height, probability

6.8.2 Python PyTorch code for maximum likelihood estimation using gradient descent

This subsection implements Gaussian maximum likelihood estimation with gradient descent, exposes its sensitivity to initialization, and shows how MAP regularization prevents parameter explosion.

• Negative log-likelihood optimization: Explain that maximizing a likelihood can be performed by minimizing its negative log-likelihood with gradient descent. State that taking the logarithm simplifies computation without changing the optimization objective's generality.
  Why it matters: This connects probabilistic parameter estimation to the loss-minimization workflow used to train machine learning models.
• Initialization sensitivity in MLE: Describe the observed dependence of gradient-descent Gaussian MLE on the starting values of μ and σ. Explain that starts far from the expected values can make the parameters grow into very large numbers rather than converge, whereas closer starts converge to the expected values.
  Why it matters: It shows why an optimization method that works in principle can fail in practice when its parameters are poorly initialized.
• MAP regularization of parameters: Explain why MAP estimation is preferred here over gradient-descent MLE: the prior p(θ) acts as a regularizer that prevents μ and σ from becoming too large. Describe the MAP loss as the negative log-likelihood augmented with μ² and σ² regularization terms.
  Why it matters: MAP introduces prior-based control over parameter values, a central idea behind regularization and Bayesian estimation.

Key terms: maximum likelihood estimation, likelihood, negative log-likelihood, gradient descent, μ, σ, MLE, parameter explosion, initial values, maximum a posteriori estimation

6.9 Gaussian mixture models

This section motivates Gaussian mixture models as weighted combinations of Gaussian components for modeling unlabeled, multimodal data with multiple classes or clusters.

• Limits of single distributions: Explain why a single simple, unimodal probability distribution cannot model the unlabeled Statsville height data or the height-weight data when the data contain multiple distinct clusters.
  Why it matters: Recognizing multimodal data tells you when a one-distribution model is inadequate.
• Gaussian mixture model definition: Describe a Gaussian mixture model as a weighted combination of a specified number of Gaussian components, used together to represent an overall probability distribution.
  Why it matters: A GMM provides a probability model for data whose overall shape is made from several bell-shaped groups.
• Components, classes, and dimensions: Relate the chosen GMM components to the problem structure: two one-dimensional Gaussian components for the two-class height example and three two-dimensional Gaussian components for the three-class height-weight example, with each component corresponding to a class.
  Why it matters: This connection lets you translate visible clusters and available features into an appropriate mixture-model structure.

Key terms: unimodal, single-peak, unlabeled data, point cloud, clusters, Gaussian mixture model, GMM, weighted combination, Gaussian components, Gaussian component

6.9.1 Probability density function of the GMM

This subsection defines a Gaussian mixture model PDF and explains how its mixture weights, means, covariances, and number of components determine its density shape and sampled clusters.

• Gaussian mixture density: Describe a GMM probability density as a weighted combination of K Gaussian component densities, with a mixture-selection probability πk for each component.
  Why it matters: The mixture PDF is the mathematical object used to model data that may come from several Gaussian-shaped groups.
• GMM parameter effects: Explain how K, πk, μk, and Σk affect a GMM PDF and its sampled point cloud: K affects the number of peaks and clusters, πk affects relative hill heights, μk sets peak and cluster-centroid locations, and Σk sets hill and cluster shapes.
  Why it matters: These parameters let you connect the algebraic definition of a GMM to the visible structure of modeled data.
• GMM probability normalization: State that the GMM density in equation 6.29 is a proper probability density: over all possible inputs in the d-dimensional space, it sums to 1.
  Why it matters: Normalization ensures that the mixture can legitimately be interpreted as a probability distribution.

Key terms: Gaussian mixture model, GMM, probability density function, Gaussian components, mixture-selection probabilities, πk, K, μk, Σk, peaks

6.9.2 Latent variables for class selection

This section interprets a Gaussian mixture model through a hidden categorical class selector, linking component weights and Gaussian densities to joint, conditional, and marginal probabilities.

• Latent class selector: Describe the latent random variable Z as a hidden categorical class selector for a Gaussian mixture model. Explain that Z=k means the kth Gaussian component, or subclass, was selected, whereas the input variable X is observed.
  Why it matters: The latent selector gives a probabilistic meaning to the otherwise hidden choice of mixture component.
• Mixture weights as priors: Explain that the mixture weight πk is the prior probability p(Z=k) of selecting component k, often modeled as Nk/N, the fraction of training examples in that subclass. State that the K weights form a categorical distribution and sum to 1.
  Why it matters: Mixture weights determine how often each class or cluster contributes to the overall data distribution.
• GMM probability decomposition: Trace how a GMM combines conditional, joint, and marginal probabilities: the kth Gaussian density is p(x|k), multiplying it by πk gives the joint probability p(x,k), and summing these joint probabilities over k gives the marginal density p(x). Describe GMM sampling as first selecting a component according to πk and then sampling from that component's Gaussian.
  Why it matters: This decomposition explains both why a GMM is a weighted sum of Gaussians and how it generates data.

Key terms: latent random variable, Z, categorical distribution, class selector, Gaussian component, mixture weights, πk, prior probability, p(Z=k), component selection probabilities

6.9.3 Classification via GMM

This section explains how a fitted Gaussian mixture model classifies an unlabeled input by assigning it to the component with the largest posterior class probability.

• GMM posterior classification: Describe how a Gaussian mixture model classifies an arbitrary input after its parameters have been fitted. State that the input is assigned to the class k with the largest posterior probability p(Z = k|X = x).
  Why it matters: This turns a fitted generative mixture model into a practical classifier for new inputs.
• Variance-sensitive cluster membership: Explain the geometric intuition behind GMM classification: posterior assignment acts like choosing a mean that is close after distance is normalized by that component's variance. Describe why a point farther from a high-variance, spread-out cluster mean can still belong to that cluster, whereas the same distance may be implausible for a tight cluster.
  Why it matters: This intuition explains why GMM decision boundaries reflect both cluster centers and cluster spreads.

Key terms: Gaussian mixture model, GMM, posterior probability, p(Z = k|X = x), latent variable, class, mean, variance, cluster, closest mean

6.9.4 Maximum likelihood estimation of GMM parameters (GMM fit)

This section derives and interprets an iterative maximum-likelihood procedure for fitting a Gaussian mixture model to unlabeled data.

• GMM fitting objective: Explain that GMM fitting estimates mixture weights, means, and covariances from unlabeled training points so that the model distribution matches the naturally clustered data as closely as possible. State that the number of components K is fixed beforehand rather than estimated by this procedure.
  Why it matters: This defines what a fitted GMM is trying to achieve and which quantities training must determine.
• Iterative MLE necessity: Explain why maximum likelihood for a GMM cannot be solved in closed form: differentiating its log-likelihood produces equations in which the unknown parameters also remain inside the right-hand-side responsibilities. Describe iterative relaxation as starting with parameter values, repeatedly recomputing updates, and stopping at a fixed point when changes or improvement are no longer appreciable.
  Why it matters: It explains why fitting a mixture requires repeated updates instead of the direct formulas available for a single Gaussian.
• Responsibility posterior meaning: Describe γik as the posterior probability that training point i belongs to component or class k under the current GMM parameters. Explain that it provides a soft, probability-weighted assignment rather than a hard class label.
  Why it matters: Responsibilities connect the probabilistic interpretation of a GMM to the parameter-update equations used to fit it.
• Responsibility-weighted updates: Describe how the iterative updates use responsibilities: Nk is the probability mass assigned to component k, πk is that mass divided by the number of data points, and μk and Σk are the responsibility-weighted mean and covariance of the training points. State that these values are recomputed using the current parameter estimates.
  Why it matters: These updates are the concrete operations that turn soft component memberships into improved GMM parameters.

Key terms: GMM, parameter set θ, mixture weights πk, means μk, covariances Σk, unlabeled training data, maximum likelihood estimation, log-likelihood, closed-form solution, iterative relaxation

6.10 Summary

This summary recaps Bayesian probability tools, uncertainty measures, and parameter-estimation methods for machine learning models.