Low-rank matrix

As explained in Generative AI in Action

A low-rank matrix is a matrix whose rank is substantially smaller than its number of rows or columns, allowing a compact representation of a transformation.

A low-rank matrix represents a compact parameter update whose rank is much smaller than the dimensions of the original weight matrix. In LoRA, these matrices adapt a pretrained language model without replacing its full architecture or changing its original weights. section 9.6.1

LoRA uses low-rank matrices to make model adaptation more efficient:

• Inserts small matrices into attention and feed-forward layers while the pretrained weight matrix remains frozen. section 9.6.1
• Trains only the adaptation parameters, reducing the computation and storage needed for task-specific fine-tuning. section 9.6.1
• Preserves most pretrained knowledge while enabling specialization, helping reduce the risk of catastrophic forgetting. section 9.6
• Stores and loads a small set of task-specific parameters alongside the pretrained model for downstream adaptations. section 9.6.1

LoRA reparameterization

A layer receives input X and uses its original pretrained weight W together with adaptation matrices A and B. Fine-tuning updates A and B, while W stays frozen. section 9.6.1

Figure 9.14 LoRA reparameterization—only A and B are trained [9]

Inspect the reparameterization diagram showing that LoRA trains only A and B while preserving W. section 9.6.1

How Generative AI in Action covers it

How deep it goes

Explains insertion, selective training, frozen weights, and runtime parameter storage without deriving the underlying matrix mathematics.

Examples

Uses LoRA’s A, B, W, and X reparameterization diagram as its concrete representation.

What it assumes

Assumes familiarity with pretrained LLM layers and weights while introducing low-rank adaptation within that architecture.

What you can do after

Apply it via LoRA to adapt an LLM by training small task-specific matrices instead of the full pretrained weight set.

How it connects

Connects low-rank matrices with LoRA, parameter-efficient fine-tuning, pretrained models, and catastrophic forgetting.

liveBookmanning.com

Read Generative AI in Action on liveBook Get the book at manning.com

Related concepts

• low-rank adaptationused by · Low-rank adaptation is a parameter-efficient fine-tuning method that adapts a model by learning low-rank updates to its weights, typically while keeping most original weights fixed.
• low-rank approximationproduced by · A low-rank approximation represents a matrix by a lower-rank matrix or factorization, often retaining important structure while reducing storage or computation.
• arraykind of · An array is an indexed, ordered collection of elements arranged in one or more dimensions, with elements accessed by position.

This explainer was generated by Manning's knowledge graph from the text of Generative AI in Action. The section links point to the parts of the book it draws on.