Low-rank approximation

As explained in LLMs in Production

A method that represents a matrix or parameter structure with a lower-rank approximation to reduce storage or computation.

A large language model can be costly to download, load into memory, and deploy. Low-rank approximation tackles part of that size problem by representing a large matrix or tensor with smaller components. It is also called low-rank factorization, low-rank decomposition, or matrix factorization. section 3.3.1 section 3.5

Figure 3.2 Example of SVD, a low-rank approximation. A is a large matrix with dimensions N and M. We can approximate it with three smaller matrices: U with dimensions M and P, Ʃ a square matrix with dimension P, and V with dimensions N and P (here we show the transpose). Usually, both P<<M and P<<N are true.

In singular value decomposition (SVD), a matrix A is approximated using U, Σ, and the transpose of V. The shared dimension P is usually much smaller than A’s dimensions; choosing a smaller P compresses more but makes the approximation less accurate. section 3.3.1

Compress a 4 × 4 matrix

A SciPy example calls `scipy.sparse.linalg.svds(matrix, k=1)` on a 4 × 4 matrix. Its returned components have shapes 4 × 1, 1, and 1 × 4: nine stored values rather than the original sixteen. Multiplying the components reconstructs a 4 × 4 approximation, not the original values exactly. section 3.3.1

SVD is one option; Tucker decomposition and canonical polyadic decomposition are also named techniques. For model compression, the practical choice is how much approximation error is acceptable for the reduction in size. Low-rank methods are also used for model adaptation through LoRA. section 3.3.1 section 3.5

How LLMs in Production covers it

How deep it goes

Explains SVD through factor dimensions and a worked rank-one example, without deriving the linear algebra.

Examples

Uses a SciPy rank-one approximation of a 4 × 4 matrix and its reconstructed values.

What it assumes

Assumes basic familiarity with matrices; introduces low-rank approximation in plain language rather than deriving SVD.

What you can do after

Apply it via SciPy’s `svds` to make a low-rank matrix approximation and inspect the size–accuracy trade-off.

How it connects

Connects SVD, Tucker decomposition, canonical polyadic decomposition, model compression, and LoRA.

liveBookmanning.com

Read LLMs in Production on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Deep Learning with R, Third EditionA low-rank decomposition represents a matrix transformation as a product of smaller matrices with an intermediate dimension lower than the original matrix dimensions.
• 

  liveBookmanning.com
  Transformers in ActionA low-rank matrix is a matrix whose rank is substantially smaller than its number of rows or columns, enabling compact parameterization.

Compare how 3 books explain low-rank approximation →

Related concepts

• low-rank adaptationused by · Low-rank adaptation is a parameter-efficient fine-tuning method that adapts a model by learning low-rank updates to its weights, typically while keeping most original weights fixed.
• DoRAused by · DoRA is a parameter-efficient fine-tuning method that decomposes model weights into separately tunable magnitude and directional components, typically applying low-rank adaptation to the directional component.
• singular value decompositionproduced by · Singular value decomposition is a matrix factorization that expresses a matrix as the product of orthogonal or unitary matrices and a diagonal matrix of nonnegative singular values.
• dropoutcontrasts with · Dropout is a neural-network regularization technique that randomly suppresses units, connections, or activations during training to reduce overfitting.

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.