Low-rank approximation
A low-rank approximation represents a matrix by a lower-rank matrix or factorization, often retaining important structure while reducing storage or computation.
LLMs in Production
A method that represents a matrix or parameter structure with a lower-rank approximation to reduce storage or computation.
- How it is used
- Reduces large-model size and supports model adaptation.
- How deep it goes
- Explains SVD through factor dimensions and a worked rank-one example, without deriving the linear algebra.
- Examples
- Uses a SciPy rank-one approximation of a 4 × 4 matrix and its reconstructed values.
- What you can do after
- Apply it via SciPy’s `svds` to make a low-rank matrix approximation and inspect the size–accuracy trade-off.
Read how LLMs in Production explains low-rank approximation →
Deep Learning with R, Third Edition
A low-rank decomposition represents a matrix transformation as a product of smaller matrices with an intermediate dimension lower than the original matrix dimensions.
- How it is used
- Reduces accelerator memory during large-language-model fine-tuning by updating low-rank attention projections instead of full kernels.
- How deep it goes
- Explains the frozen-plus-update construction, parameter arithmetic, implementation, and expressive-power trade-off.
- Examples
- Uses a 2,048 × 2,048 kernel with rank 8, an R layer implementation, and KerasHub configuration for Gemma.
- What you can do after
- Operate/configure LoRA through KerasHub and selectively enable rank-8 adapters on attention layers.
Read how Deep Learning with R, Third Edition explains low-rank approximation →
Transformers in Action
A low-rank matrix is a matrix whose rank is substantially smaller than its number of rows or columns, enabling compact parameterization.
- How it is used
- Reduces trainable parameters when adapting pretrained transformer weights to downstream tasks.
- How deep it goes
- Defines factor dimensions, demonstrates SVD reconstruction, and extends the method to quantized decomposition without deriving optimization internals.
- Examples
- Uses a random 6×6 matrix, SVD-derived factors, and an Unsloth configuration targeting Qwen3-4B projection modules.
- What you can do after
- Apply it via Unsloth by selecting a rank and attaching low-rank adapters to transformer projection modules.
Read how Transformers in Action explains low-rank approximation →
Related concepts
- low-rank adaptationused by · Low-rank adaptation is a parameter-efficient fine-tuning method that adapts a model by learning low-rank updates to its weights, typically while keeping most original weights fixed.
- DoRAused by · DoRA is a parameter-efficient fine-tuning method that decomposes model weights into separately tunable magnitude and directional components, typically applying low-rank adaptation to the directional component.
- singular value decompositionproduced by · Singular value decomposition is a matrix factorization that expresses a matrix as the product of orthogonal or unitary matrices and a diagonal matrix of nonnegative singular values.
- dropoutcontrasts with · Dropout is a neural-network regularization technique that randomly suppresses units, connections, or activations during training to reduce overfitting.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.