Singular value decomposition (SVD) factors any m × n matrix, square or rectangular, as A = UΣVᵀ. U and V are orthogonal; Σ is diagonal and orders singular values from largest to smallest. This factorization makes a general matrix easier to analyze and compute with. section 4.5 section 4.7
Equation 4.11
| A = UΣVT |
The central SVD factorization separates a matrix into two orthogonal changes of basis and a diagonal scaling matrix. section 4.5
The right singular vectors are eigenvectors of AᵀA and fill V; the left singular vectors are eigenvectors of AAᵀ and fill U. Both AᵀA and AAᵀ are symmetric positive semidefinite, so their eigenvalues are nonnegative and their eigenvectors are orthogonal. For a nonzero eigenvalue λᵢ of AᵀA, the matching left singular vector is obtained by scaling Av̂ᵢ by 1/√λᵢ. section 4.5.1 section 4.5.2
SVD exposes useful matrix structure:
- Rank equals the number of nonzero singular values, linking degeneracy to linear dependence among rows or columns. section 4.5.5
- Extra singular-vector columns in a rectangular factorization can vanish because their corresponding entries in Σ are zero. section 4.5.2
- Large and small singular values distinguish high-variation directions from weaker variation that may be noise. section 4.6.4 section 4.7
To solve A x = b, first rotate b with Uᵀ, divide by the diagonal entries of Σ, then rotate back with V. For invertible square matrices this agrees with inverse-based solving; for nonsquare systems it yields the Moore-Penrose inverse solution, described as the best-effort approximation. The diagonal and orthogonal factors make these substeps simple, while avoiding direct inversion of A. section 4.5.4 section 4.5.6
Solving systems in PyTorch
`torch.linalg.svd(A)` returns `U`, singular values `S`, and `V_t`. Form a diagonal inverse from `1 / S`, then compute `V_t.T @ (S_inv @ (U.T @ b))`. An invertible 3 × 3 test system produces `[1.0, 2.0, 3.0]` through both matrix inversion and SVD; an overdetermined system likewise matches `torch.linalg.pinv(A)`. section 4.5.6
For mean-subtracted data matrix X with instances as rows, XᵀX is its covariance matrix. Because V contains eigenvectors of XᵀX, SVD computes PCA: columns of V give principal vectors, while diagonal values in S give principal values. Retaining directions with larger variation supports lower-dimensional, relatively noise-free representations. section 4.5.3 section 4.5.7 section 4.7
SVD also supports latent semantic analysis (LSA). Applied to a document-term matrix, it identifies principal directions interpreted as topics. Projecting documents onto topic axes can capture indirect similarity, such as an association between documents using “gun” and “violence” even when they share no term directly. section 4.6.2 section 4.6.4
