Singular value decomposition

As explained in LLMs in Production

Singular value decomposition is a matrix factorization that expresses a matrix as a product of orthogonal and diagonal matrices.

Singular value decomposition (SVD) factors a large matrix into three smaller matrices—U, Σ, and V—to create a lower-dimensional approximation. The trade-off is practical: shrinking Σ reduces parameters and memory, but also reduces approximation accuracy. section 3.3.1

SVD becomes useful when compression or parameter-efficient adaptation matters:

• Compresses matrices or tensors by retaining a lower-dimensional representation rather than every original parameter. section 3.3.1
• Approximates a 100 × 100 weight update as Wa × Wb, using a rank c smaller than 100. section 5.3.5
• Trades memory savings against task complexity and accuracy: higher rank preserves more complexity, while lower rank saves more memory. section 5.3.5

A 4 × 4 low-rank approximation

Using SciPy and NumPy, decompose a 4 × 4 matrix with svds(matrix, k=1), then reconstruct it by multiplying u, s, and vt. The reconstruction uses 9 parameters instead of 16, but its values only approximate the original matrix. section 3.3.1

Figure 3.2 Example of SVD, a low-rank approximation. A is a large matrix with dimensions N and M. We can approximate it with three smaller matrices: U with dimensions M and P, Ʃ a square matrix with dimension P, and V with dimensions N and P (here we show the transpose). Usually, both P<<M and P<<N are true.

View the dimensional relationship between the original matrix A and its smaller U, Σ, and V factors. section 3.3.1

How LLMs in Production covers it

How deep it goes

Defines U, Σ, and V, demonstrates reconstruction with NumPy and SciPy, and applies rank selection to LoRA without deriving SVD algorithms.

Examples

Uses a 4 × 4 NumPy matrix, SciPy svds reconstruction, and a 100 × 100 LoRA weight update with rank two.

What it assumes

Starts with large matrices and basic SciPy and NumPy usage while leaving complete backpropagation and rank algorithms unexplained.

What you can do after

Apply it via SciPy and NumPy to reconstruct a low-rank matrix and reason about LoRA rank, memory, and accuracy trade-offs.

How it connects

Connects low-rank approximation, matrix factorization, LoRA, PEFT, model weights, compression, and approximation accuracy.

liveBookmanning.com

Read LLMs in Production on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Data Without LabelsSingular value decomposition is a matrix factorization that expresses a matrix as the product of two orthogonal matrices and a diagonal matrix of singular values.
• 

  liveBookmanning.com
  Math and Architectures of Deep LearningThe statement that any matrix can be decomposed as a product of orthogonal matrices and a diagonal matrix in singular value decomposition form.
• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionSingular value decomposition is a matrix factorization that expresses a matrix as the product of orthogonal singular-vector matrices and a diagonal singular-value matrix.
• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIA matrix factorization that expresses a matrix as the product of orthogonal matrices and a diagonal matrix of singular values.
• 

  liveBookmanning.com
  Grokking Machine LearningA matrix decomposition that expresses a matrix as the product of orthogonal matrices and a diagonal matrix of singular values.

Compare how 6 books explain singular value decomposition →

Related concepts

• principal component analysisused by · Principal component analysis is a linear dimensionality-reduction technique that transforms data into orthogonal components ordered by the variance they capture.
• singular valueproduces · A singular value is a nonnegative scalar in a matrix’s singular value decomposition that measures stretching along a corresponding singular direction.
• co-occurrence matrixuses · A co-occurrence matrix is a table whose entries record the frequency or weight with which pairs of items occur together within specified contexts.
• dimensionality reductionuses · Dimensionality reduction is the transformation of data into a representation with fewer dimensions while retaining selected information, structure, or relationships.
• truncated singular value decompositionincludes · Truncated singular value decomposition is a matrix factorization technique that approximates a matrix using a selected number of singular components.
• compressionuses · Compression is the transformation of information into a more compact representation to reduce storage or transmission requirements while retaining information needed for recovery or use.

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.