A projection replaces a point or vector with its representation on a chosen line, plane, or subspace. Geometrically, orthogonal projection finds the closest point in that subspace: draw a perpendicular from the original point, and its landing point is the projection. The remaining difference, called the residual, is perpendicular to every vector in the subspace. section 2.12.5
Projection is useful when data has many coordinates but its important variation lies near a lower-dimensional shape. Points near the X–Y plane, for example, can be projected onto that plane and represented in two dimensions. This discards small deviations away from the subspace while retaining the larger variation within it. section 4.1 section 4.4.2
The same idea appears in several related forms:
- Least-squares solving seeks A x in the column space of A that is closest to a target y. Orthogonality of the residual gives Aᵀ(y − A x) = 0, leading to the Moore–Penrose pseudo-inverse solution. section 2.12.5
- PCA projects data onto principal axes with large principal values. Removing axes with small values converts, for example, a nearly one-dimensional 2D cloud into a 1D representation. section 4.4.2
- A dot product l̂ᵀx is the scalar component, or projection, of x along unit direction l̂. Variance of these scalar projections measures a random vector’s spread in different directions. section 5.7.2
- A projection matrix can map collinear vectors in 3D to collinear vectors in 2D; this preservation of collinearity is a defining behavior of linear transforms represented by matrix multiplication. section 2.10
Compressing data with PCA
Suppose 2D points form a long, narrow cloud around a line. PCA identifies the line as the direction of greatest spread. Replacing every point by its projection on that principal axis stores one coordinate instead of two, preserves the main pattern, and drops the small perpendicular deviations as noise. Retaining only the first few principal axes similarly compresses larger multidimensional datasets. section 4.4.2 section 4.4.5
Projection can also produce useful feature spaces rather than merely fewer coordinates. In latent semantic analysis, document-term vectors are multiplied by retained topic vectors to form lower-dimensional document-topic vectors. Cosine similarity in this topic space can capture indirect relationships not evident in the original term space. section 4.6.2
Figure 4.3 Dimensionality reduction via PCA. Original data points are shown with filled little circles, and hollow circles represent lower-dimensional representations.
|
(a) Dimensionality reduction from 2D to 1D |
|
(b) Dimensionality reduction from 3D to 2D |
Compare original points with their projected 1D and 2D representations, and see the small perpendicular deviations that PCA removes. section 4.4.2
