liveBook · Manning

Low-rank approximation

Also called low-rank decomposition

A low-rank approximation represents a matrix or transformation with a lower-rank matrix or factorization, often retaining dominant structure while reducing storage or computation.

How 3 Manning books explain it

LLMs in Production

A method that represents a matrix or parameter structure with a lower-rank approximation to reduce storage or computation.

How it is used
Uses low-rank approximation to shrink large models that are costly to load and deploy.
How deep it goes
Works through rank-one SVD and its compression–accuracy trade-off, while naming Tucker and canonical polyadic decompositions without explaining them.
Examples
Compresses a 4 × 4 matrix into rank-one factors and reconstructs an approximate matrix.
What you can do after
Apply it via SciPy to obtain a low-rank matrix approximation.

Read how LLMs in Production explains low-rank approximation →

Deep Learning with R, Third Edition

A low-rank decomposition represents a matrix transformation as a product of smaller matrices with an intermediate dimension lower than the original matrix dimensions.

How it is used
Reduces accelerator memory during large-language-model fine-tuning by updating low-rank attention projections instead of full kernels.
How deep it goes
Explains the frozen-plus-update construction, parameter arithmetic, implementation, and expressive-power trade-off.
Examples
Uses a 2,048 × 2,048 kernel with rank 8, an R layer implementation, and KerasHub configuration for Gemma.
What you can do after
Operate/configure LoRA through KerasHub and selectively enable rank-8 adapters on attention layers.

Read how Deep Learning with R, Third Edition explains low-rank approximation →

Transformers in Action

A low-rank matrix is a matrix whose rank is substantially smaller than its number of rows or columns, enabling compact parameterization.

How it is used
Reduces trainable parameters when adapting pretrained transformer weights to downstream tasks.
How deep it goes
Defines factor dimensions, demonstrates SVD reconstruction, and extends the method to quantized decomposition without deriving optimization internals.
Examples
Uses a random 6×6 matrix, SVD-derived factors, and an Unsloth configuration targeting Qwen3-4B projection modules.
What you can do after
Apply it via Unsloth by selecting a rank and attaching low-rank adapters to transformer projection modules.

Read how Transformers in Action explains low-rank approximation →

Also covered in

What to know first

Related concepts

Questions about low-rank approximation

What is low-rank approximation?

A low-rank approximation represents a matrix or transformation with a lower-rank matrix or factorization, often retaining dominant structure while reducing storage or computation.

Which Manning books explain low-rank approximation?

LLMs in Production, Deep Learning with R, Third Edition and Transformers in Action.

What should I know before learning low-rank approximation?

Manning books that teach low-rank approximation usually introduce fine-tuning, low-rank adaptation, quantization and attention first.

What is low-rank approximation a kind of?

Low-rank approximation is a kind of approximation. Approximation is representing or computing a result that is sufficiently close to an exact result for an intended purpose, without requiring exact equality.

What are the parts of low-rank approximation?

Low-rank approximation includes projection matrix.

What is low-rank approximation also called?

Low-rank approximation is also called low-rank decomposition.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.