Low-rank approximation
A low-rank approximation represents a matrix or transformation with a lower-rank matrix or factorization, often retaining dominant structure while reducing storage or computation.
LLMs in Production
A method that represents a matrix or parameter structure with a lower-rank approximation to reduce storage or computation.
- How it is used
- Uses low-rank approximation to shrink large models that are costly to load and deploy.
- How deep it goes
- Works through rank-one SVD and its compression–accuracy trade-off, while naming Tucker and canonical polyadic decompositions without explaining them.
- Examples
- Compresses a 4 × 4 matrix into rank-one factors and reconstructs an approximate matrix.
- What you can do after
- Apply it via SciPy to obtain a low-rank matrix approximation.
Read how LLMs in Production explains low-rank approximation →
Deep Learning with R, Third Edition
A low-rank decomposition represents a matrix transformation as a product of smaller matrices with an intermediate dimension lower than the original matrix dimensions.
- How it is used
- Reduces accelerator memory during large-language-model fine-tuning by updating low-rank attention projections instead of full kernels.
- How deep it goes
- Explains the frozen-plus-update construction, parameter arithmetic, implementation, and expressive-power trade-off.
- Examples
- Uses a 2,048 × 2,048 kernel with rank 8, an R layer implementation, and KerasHub configuration for Gemma.
- What you can do after
- Operate/configure LoRA through KerasHub and selectively enable rank-8 adapters on attention layers.
Read how Deep Learning with R, Third Edition explains low-rank approximation →
Transformers in Action
A low-rank matrix is a matrix whose rank is substantially smaller than its number of rows or columns, enabling compact parameterization.
- How it is used
- Reduces trainable parameters when adapting pretrained transformer weights to downstream tasks.
- How deep it goes
- Defines factor dimensions, demonstrates SVD reconstruction, and extends the method to quantized decomposition without deriving optimization internals.
- Examples
- Uses a random 6×6 matrix, SVD-derived factors, and an Unsloth configuration targeting Qwen3-4B projection modules.
- What you can do after
- Apply it via Unsloth by selecting a rank and attaching low-rank adapters to transformer projection modules.
Read how Transformers in Action explains low-rank approximation →
Also covered in
- LLM Customization and Fine-TuningIn early access (MEAP)
- Sutskever's List
- Rearchitecting LLMsIn early access (MEAP)
What to know first
- fine-tuningFine-tuning is the continued training of a pretrained model on targeted data or objectives to adapt its parameters or behavior to a task, domain, or use.
- low-rank adaptationLow-rank adaptation (LoRA) is a parameter-efficient fine-tuning technique that adapts a model by learning low-rank updates to its weights, typically while keeping most original weights fixed.
- quantizationModel quantization represents model parameters or computations with reduced-precision numerical values, often to lower storage or computational requirements.
- attentionAttention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
Related concepts
- low-rank adaptationused by · Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning technique that adapts a model by learning low-rank updates to its weights, typically while keeping most original weights fixed.
- approximationkind of · Approximation is representing or computing a result that is sufficiently close to an exact result for an intended purpose, without requiring exact equality.
- DoRAused by · DoRA is a parameter-efficient fine-tuning method that decomposes model weights into separately tunable magnitude and directional components, typically applying low-rank adaptation to the directional component.
- singular value decompositionproduced by · Singular value decomposition is a matrix factorization that expresses a matrix using orthogonal or unitary matrices and a diagonal matrix of nonnegative singular values.
Questions about low-rank approximation
What is low-rank approximation?
A low-rank approximation represents a matrix or transformation with a lower-rank matrix or factorization, often retaining dominant structure while reducing storage or computation.
Which Manning books explain low-rank approximation?
LLMs in Production, Deep Learning with R, Third Edition and Transformers in Action.
What should I know before learning low-rank approximation?
Manning books that teach low-rank approximation usually introduce fine-tuning, low-rank adaptation, quantization and attention first.
What is low-rank approximation a kind of?
Low-rank approximation is a kind of approximation. Approximation is representing or computing a result that is sufficiently close to an exact result for an intended purpose, without requiring exact equality.
What are the parts of low-rank approximation?
Low-rank approximation includes projection matrix.
What is low-rank approximation also called?
Low-rank approximation is also called low-rank decomposition.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


