How deeply does Math and Architectures of Deep Learning cover dimensionality reduction?

Derives geometric PCA intuition, recaps its SVD workflow, and implements projection and approximate reconstruction in PyTorch.

Dimensionality reduction

As explained in Math and Architectures of Deep Learning

The reduction of the effective number of dimensions in a representation by a transformation such as a singular weight matrix.

Dimensionality reduction replaces a high-dimensional representation with a lower-dimensional one when the data lies near a line, plane, or other lower-dimensional subspace. The approximation discards small deviations from that subspace, yielding a more compact representation that can expose broad patterns, lower storage costs, and often suppress noise. section 4.1 section 5.7.2

PCA performs this reduction for data concentrated near a line or hyperplane. Its principal axes are ordered by variance: retain axes with large principal values and drop axes with small ones. Projecting points onto the retained axes preserves the main spread while losing the small orthogonal variations. section 4.4.2 section 4.7

Figure 14.3 The original 3D data in figure 14.3a shows high correlation: points are clustered around the X₀ = X₂ plane. The first principal component corresponds to the direction of maximum variance. The last (third) principal reduced to a 2D latent vector.

(a) Original 3D data

(b) Lower-dimensional 2D representation obtained by setting the third principal value to zero

(c) The principal vectors of the original data. The third principal vector is normal to X₀ = X₂ plane; the other two are in-plane.

The 3D-to-2D picture makes the criterion visible: a small third principal value identifies the direction normal to the data plane as expendable. section 14.4

A PCA reduction can be carried out as follows:

Arrange instances as rows of a data matrix and subtract the mean vector to center the data. section 14.4
Compute an SVD of the centered matrix; its right singular vectors provide the principal directions. section 4.7
Keep the vectors for the desired large principal values and multiply the data by them to obtain lower-dimensional coordinates. section 14.4.1
Optionally pad the reduced coordinates with zeros, multiply back by the SVD factor, and restore the mean to reconstruct an approximation. section 14.4.1

Correlated document terms

If document features for “gun” and “violence” have similar values, their points cluster near the line x = y. Replacing the two correlated axes with that single direction simplifies the representation and can reveal a gun-violence topic. section 4.1

PyTorch projection and reconstruction

For 1,000 three-dimensional points near the X0 = X2 plane, center X, run torch.linalg.svd, retain the first two columns of V, and compute X_proj = X @ V_trimmed. Reconstructing after restoring the omitted coordinate is approximate because reduction has discarded information. section 14.4.1

Dimensionality reduction also frames learned latent representations. An autoencoder maps an observed input to a usually smaller latent vector, then is trained to reconstruct the input; the dimension limit forces retention of features useful to the goal and omission of others. A neural-network linear layer with a zero-determinant weight matrix likewise effectively reduces dimensionality. section 14.0 section 12.3

How Math and Architectures of Deep Learning covers it

Cover of Math and Architectures of Deep Learning

liveBook manning.com

Read Math and Architectures of Deep Learning on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Data Without LabelsDimensionality reduction is the transformation of data into fewer dimensions while retaining important structure or information.
liveBook manning.com

Natural Language Processing in Action, Second EditionDimension reduction is the transformation of data into a representation with fewer features or dimensions while retaining relevant information.
liveBook manning.com

Outlier Detection in PythonDimensionality reduction is the transformation of data into fewer dimensions while retaining relevant information.
liveBook manning.com

Grokking Machine LearningA technique that simplifies data by reducing or combining features while preserving as much information as possible.
liveBook manning.com

Graph Neural Networks in ActionDimensionality reduction is the transformation of data into a representation with fewer numerical dimensions while retaining selected information or structure.
liveBook manning.com

Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIDimensionality reduction transforms data with many variables into a lower-dimensional representation while attempting to preserve important structure or information.
liveBook manning.com

Cyber Threat HuntingDimensionality reduction transforms data into a representation with fewer features while retaining important information.
liveBook manning.com

Graph Algorithms for Data ScienceDimensionality reduction is a technique that represents data with fewer numerical dimensions while preserving selected structure or information.
liveBook manning.com

Delivering Machine Learning Projects: From design to deploymentDimensionality reduction is the transformation of data into fewer dimensions while attempting to preserve informative structure.
liveBook manning.com

Inside Deep Learning: Math, Algorithms, ModelsThe process of mapping data from a higher-dimensional space into a lower-dimensional representation while retaining important structure.

Compare how 11 books explain dimensionality reduction →

Related concepts

principal component analysisincludes · Principal component analysis is a linear dimensionality-reduction technique that transforms data into orthogonal components ordered by the variance they capture.
UMAPincludes · UMAP, or Uniform Manifold Approximation and Projection, is a nonlinear manifold-learning technique that embeds high-dimensional data in a lower-dimensional space while preserving important neighborhood and structural relationships.
t-SNEincludes · t-SNE is a nonlinear dimensionality-reduction technique that maps high-dimensional data to a lower-dimensional space while emphasizing local similarities or neighborhood relationships.
curse of dimensionalityaddresses · The curse of dimensionality is the set of statistical, geometric, and computational difficulties that tend to increase as the number of dimensions grows.
embeddinguses · A numerical vector representation of an item or input that encodes features or relationships useful for computational tasks.
unsupervised learningkind of · Unsupervised learning is a machine-learning approach that discovers patterns, structure, or representations in data without labeled target outputs.

This explainer was generated by Manning's knowledge graph from the text of Math and Architectures of Deep Learning. The section links point to the parts of the book it draws on.