A principal axis is a direction in multidimensional data along which variance is measured. For a unit direction l̂, the projected variance is l̂ᵀCl̂, where C is the covariance matrix. This variance is maximal when l̂ is the eigenvector associated with C’s largest eigenvalue; that direction is the first principal axis. Further principal axes follow eigenvectors ordered by decreasing eigenvalues and are mutually orthogonal. section 4.4.1 section 4.7
Geometrically, principal axes reveal the directions in which a point cloud spreads most. Projecting data onto axes with large principal values preserves its dominant structure, while discarding directions with small values can remove small perturbations interpreted as noise. section 4.4.2 section 4.7
From a tilted cloud to fewer dimensions
A two-dimensional dataset concentrated around a slanted line has one large principal value and one small one. Its first principal axis follows the line of maximum spread. Projecting each point onto that axis converts the data to one dimension while retaining the main pattern and reducing storage. Likewise, a three-dimensional cloud concentrated around a plane can be projected onto its first two principal axes. section 4.4.2
Figure 4.3 Dimensionality reduction via PCA. Original data points are shown with filled little circles, and hollow circles represent lower-dimensional representations.
|
(a) Dimensionality reduction from 2D to 1D |
|
(b) Dimensionality reduction from 3D to 2D |
Compare the original points with their lower-dimensional projections for 2D-to-1D and 3D-to-2D reductions. section 4.4.2
Principal axes also provide a compression basis: retaining projections on the first few axes reduces the number of components stored, while omitting components associated with smaller principal values causes comparatively little loss. section 4.4.5
