How deeply does Deep Learning with R, Third Edition cover low-rank approximation?

Explains the frozen-plus-update construction, parameter arithmetic, implementation, and expressive-power trade-off.

Low-rank approximation

As explained in Deep Learning with R, Third Edition

A low-rank decomposition represents a matrix transformation as a product of smaller matrices with an intermediate dimension lower than the original matrix dimensions.

Represents a large matrix transformation as two smaller projections with a narrow intermediate dimension. In LoRA, the original kernel stays frozen while the low-rank update learns task-specific changes with far fewer trainable parameters. section 16.3.3

A 2,048 × 2,048 projection with rank 8

A full kernel contains 4,194,304 parameters. Replacing its trainable update with projections of shapes 2,048 × 8 and 8 × 2,048 uses 32,768 parameters. The narrow rank-8 middle limits expressive power, but can be sufficient for fine-tuning after pretraining. The resulting transformation combines the frozen projection with the learned update. section 16.3.3

LoRA applies the approximation operationally by:

Freezing most Transformer parameters and adding low-rank weights to attention projections. section 16.3.3
Reducing trainable storage, gradients, and optimizer-state memory while retaining the full model parameters. section 16.3.3

How Deep Learning with R, Third Edition covers it

Cover of Deep Learning with R, Third Edition

liveBook manning.com

Read Deep Learning with R, Third Edition on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

LLMs in ProductionA method that represents a matrix or parameter structure with a lower-rank approximation to reduce storage or computation.
liveBook manning.com

Transformers in ActionA low-rank matrix is a matrix whose rank is substantially smaller than its number of rows or columns, enabling compact parameterization.

Compare how 3 books explain low-rank approximation →

Related concepts

low-rank adaptationused by · Low-rank adaptation is a parameter-efficient fine-tuning method that adapts a model by learning low-rank updates to its weights, typically while keeping most original weights fixed.
DoRAused by · DoRA is a parameter-efficient fine-tuning method that decomposes model weights into separately tunable magnitude and directional components, typically applying low-rank adaptation to the directional component.
singular value decompositionproduced by · Singular value decomposition is a matrix factorization that expresses a matrix as the product of orthogonal or unitary matrices and a diagonal matrix of nonnegative singular values.
dropoutcontrasts with · Dropout is a neural-network regularization technique that randomly suppresses units, connections, or activations during training to reduce overfitting.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with R, Third Edition. The section links point to the parts of the book it draws on.