How deeply does Math and Architectures of Deep Learning cover S / ATA / A^T A / A A^T?

Derives symmetry, positive semidefiniteness, eigenvector transfer, and SVD construction, but does not provide numerical computation.

S / ATA / A^T A / A A^T

As explained in Math and Architectures of Deep Learning

The matrix product of A transpose and A, used here as the matrix whose eigenvalues are the singular values mentioned in the section.

For a matrix A, ATA converts questions about how A amplifies unit vectors into an eigenvalue problem. Its largest eigenvalue determines the maximum amplification associated with the spectral norm. section 4.3.1

If A has m rows and n columns, ATA is an n × n symmetric positive-semidefinite matrix. Its eigenvalues are non-negative, and its eigenvectors are orthogonal; AAT has the corresponding properties on the output side. section 4.5.1

The central relationships are:

||Ax̂||² = x̂TATAx̂ for a unit vector x̂. section 4.3.1
The eigenvalues of ATA determine the singular-value structure used in the SVD construction. section 4.5.1 section 4.5.2
Applying A to an eigenvector of ATA produces an eigenvector of AAT after normalization by the square root of its eigenvalue. section 4.5.1

From ATA to SVD

Let (λi, v̂i) be an eigenvalue–eigenvector pair of ATA. The vector ûi = Av̂i/√λi becomes an eigenvector of AAT with eigenvalue λi. These paired eigenvectors form the ingredients for A = UΣVT. section 4.5.1 section 4.5.2

||Ax̂|| = (Ax̂)^T(Ax̂) = x̂^TA^TAx̂

Use the equation to connect vector amplification, matrix multiplication, and the quadratic form x̂TATAx̂. section 4.3.1

How Math and Architectures of Deep Learning covers it

Cover of Math and Architectures of Deep Learning

liveBook manning.com

Read Math and Architectures of Deep Learning on liveBook Get the book at manning.com

Related concepts

singular valueequivalent to · A singular value is a nonnegative scalar in a matrix’s singular value decomposition that measures stretching along a corresponding singular direction.
eigenvalueproduces · An eigenvalue is a scalar by which a linear transformation scales a corresponding nonzero eigenvector without changing its direction.
eigenvectorproduces · An eigenvector is a nonzero vector that a linear transformation maps to a scalar multiple of itself.
singular vectorproduces · A singular vector is a vector in a matrix’s singular value decomposition belonging to an orthonormal left or right singular-vector basis.

This explainer was generated by Manning's knowledge graph from the text of Math and Architectures of Deep Learning. The section links point to the parts of the book it draws on.