A covariance matrix summarizes the spread of multidimensional data around its mean. In one dimension, variance is the average squared distance from the mean. In two or more dimensions, spread depends on direction, so one scalar is not enough. The covariance matrix C packages the variances along coordinate axes and the joint variation between coordinates into a single matrix. section 4.4 section 5.7.2
For data vectors x⁽ⁱ⁾ with mean vector μ, construct C by averaging the outer products (x⁽ⁱ⁾ − μ)(x⁽ⁱ⁾ − μ)ᵀ. Its diagonal entries are coordinate-wise variances, such as σ₀₀ and σ₁₁. Its off-diagonal entries are cross-covariances, which record joint variation across coordinate directions. This is an outer product, not a dot product: each centered data vector contributes a matrix rather than a scalar. section 4.4
| The covariance of a multidimensional point distribution is a matrix that allows us to easily measure the spread or packing density in any desired direction. It also allows us to easily figure out the direction in which the maximum spread occurs and what that spread is. |
Use the matrix as a directional-spread operator: it supports measuring how tightly a multidimensional point cloud is packed along any chosen direction. section 5.7.2
Given a unit direction l̂, project every data point onto that direction. The variance of those projected scalar values is the quadratic form l̂ᵀCl̂. Thus C does more than describe variation along the original axes: it measures spread in every possible direction through a matrix-vector calculation. section 4.4.1 section 4.7
The directions of greatest and least spread are encoded by C’s eigensystem. The maximum of l̂ᵀCl̂ occurs when l̂ is the eigenvector with C’s largest eigenvalue. This eigenvector is the first principal axis, and its eigenvalue is the variance along that axis. Subsequent eigenvectors, ordered by decreasing eigenvalue, give orthogonal principal axes and their remaining spreads. section 4.4.1 section 4.7
Mean-centered data and PCA
Arrange mean-subtracted data vectors as rows of a matrix X. Then XᵀX yields the covariance matrix in this setup. Its eigenvalues and eigenvectors give the principal components; equivalently, applying singular value decomposition to X performs PCA after mean subtraction. Keeping directions with larger eigenvalues produces a lower-dimensional representation that can preserve dominant patterns while discarding lower-variance variation. section 4.5.3 section 4.7
For a multivariate Gaussian, the covariance matrix Σ controls the shape of the probability density’s base and its sample cloud. Equal diagonal values with no cross-covariance give a circular base; more general covariance produces an ellipse whose axes align with Σ’s eigenvectors. In Gaussian mixture models, each component covariance Σₖ controls the shape of that component’s hill and cluster. section 5.10 section 6.9.1
PyTorch multivariate Gaussian
A covariance matrix can be supplied directly when constructing a multivariate normal distribution. For example, C = [[5.0, 0.0], [0.0, 5.0]] with mean [0.0, 0.0] defines a bivariate Gaussian in torch.distributions.MultivariateNormal. The distribution uses C to evaluate multivariate log probability, and sampled data can be checked against its reported mean and variance. section 5.9.2
Covariance matrices also parameterize stochastic latent representations. A Gaussian VAE encoder can emit μ(x) and Σ(x), defining a latent distribution from which a latent vector is sampled during training. Small covariance norm corresponds to a narrow distribution and a tight cloud of latent samples around the mean. section 14.7.1
