How deeply does Math and Architectures of Deep Learning cover Gaussian mixture model?

Defines component parameters and latent selection, then develops posterior classification and iterative maximum-likelihood fitting.

Gaussian mixture model

As explained in Math and Architectures of Deep Learning

A probabilistic model representing a distribution as a weighted combination of Gaussian components.

A Gaussian mixture model (GMM) represents an unlabeled, multi-cluster distribution as a weighted combination of Gaussian components. It is useful when one Gaussian cannot match the overall point cloud but each natural cluster can be approximated by a Gaussian. A component can correspond to a class, allowing the fitted distribution to generate similar samples or assign inputs to clusters. section 6.0 section 6.9

A GMM has K components with mixture weights πk, means μk, and covariances Σk. K controls the number of peaks or clusters; πk controls their relative heights; μk sets peak locations and cluster centroids; and Σk controls the shape and spread. The weights form probabilities over components, so they combine the component densities into one proper distribution. section 6.9.1 section 6.9.2

Height and weight clusters

Unlabeled height data for adult females and adult males can be modeled with two one-dimensional Gaussians. Unlabeled height-and-weight data for adult females, adult males, and children can be modeled with three two-dimensional Gaussians. Their combined density has separate bell-shaped peaks even though the observed data carries no class labels. section 6.9

Figure 6.8 Various GMMs (solid curves) with the same Gaussian components (dotted and dashed curves, respectively) but different π₁ and π₂ values

(a) Gaussian components μ₁ = 152, μ₂ = 175, σ₁ = σ₂ = 9

(b) GMM with π₁ = 0.5, π₂ = 0.5

(d) GMM with π₁ = 0.3, π₂ = 0.7

Compare mixtures built from the same two Gaussian components while changing π1 and π2 to see how weights alter the combined density's peak heights. section 6.9.1

A latent categorical variable Z selects a component. Z = k means that the kth Gaussian class was selected; its prior probability is πk, while the observed input is drawn from that component's Gaussian density. Marginalizing over Z yields the overall GMM density. section 6.9.2

A fitted GMM supports classification and parameter estimation:

For an input x, compute the posterior probability of every component and assign x to the component with the largest p(Z = k | X = x). This resembles choosing the closest mean while accounting for each cluster's variance. section 6.9.3
Fit πk, μk, and Σk from unlabeled points by maximum likelihood. Unlike a single Gaussian, a GMM has no closed-form MLE here, so parameters are iteratively updated through E-steps and M-steps until likelihood stops increasing. section 6.9.4

How Math and Architectures of Deep Learning covers it

Cover of Math and Architectures of Deep Learning

liveBook manning.com

Read Math and Architectures of Deep Learning on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Outlier Detection in PythonA Gaussian mixture model represents a probability distribution as a weighted combination of multiple Gaussian component distributions.
liveBook manning.com

Data Without LabelsA probabilistic model that represents data as a weighted combination of Gaussian probability distributions.

Compare how 3 books explain Gaussian mixture model →

Related concepts

soft clusteringuses · Soft clustering is a clustering technique that assigns each item graded or probabilistic membership across multiple clusters rather than exclusive membership in one.
generative modelingkind of · Generative modeling is a machine-learning approach that learns patterns or a data distribution to generate new samples resembling the data.
normal distributionuses · A normal distribution is a continuous probability distribution with a symmetric, bell-shaped density determined by its mean and variance.
expectation maximizationused by · Expectation maximization is an iterative algorithm for estimating model parameters with latent or unobserved variables by alternating latent-variable estimation with parameter optimization.
K-means clusteringcontrasts with · K-means clustering partitions data into a specified number of groups by assigning observations to representative centers and iteratively updating those centers to reduce within-group distances.

This explainer was generated by Manning's knowledge graph from the text of Math and Architectures of Deep Learning. The section links point to the parts of the book it draws on.