How deeply does Transformers in Action cover low-rank adaptation?

Explains frozen weights, factor dimensions, rank constraints, SVD intuition, and library configuration, but not the training loop.

Low-rank adaptation

As explained in Transformers in Action

A low-rank adapter is a parameter-efficient fine-tuning component that represents a weight update through low-rank factors.

Low-rank adaptation (LoRA) fine-tunes a pretrained transformer without updating its original weights. It learns a compact update for selected layers, reducing trainable parameters while preserving the pretrained model. section 8.3 section 8.3.1

For an original weight matrix W₀, LoRA represents the task-specific update ΔW as BA, giving W₀ + ΔW = W₀ + BA. With B ∈ ℝᵈˣʳ and A ∈ ℝʳˣᵏ, the rank r is smaller than both matrix dimensions, so the update remains low-rank. section 8.3.1

LoRA’s practical design choices include:

Freeze W₀ and train only the factor matrices A and B. section 8.3.1
Insert adapters into selected transformer layers, including attention projections and feed-forward projections. section 8.3.1
Treat rank as a hyperparameter: larger ranks can represent richer updates, while smaller ranks emphasize parameter efficiency. section 8.3.1
Use LoRA as a baseline, then compare DoRA, QLoRA, QA-LoRA, or LQ-LoRA when convergence, memory, or quantization constraints dominate. section 8.3.7

Illustrative matrix decomposition

A simplified NumPy example uses singular value decomposition to construct B and A whose product approximates a matrix W. This demonstrates rank-r approximation; actual LoRA training learns B and A rather than computing them with SVD. section 8.3.1

Listing 8.11 Illustrative example of LoRA

def lora_decomposition(W, r):
    B = np.zeros((W.shape[0], r)) #1
    A = np.random.normal(0, 1, (r, W.shape[1]))

    U, S, Vt = np.linalg.svd(W, full_matrices=False) #2
    B[:, :r] = U[:, :r] * np.sqrt(S[:r])
    A[:r, :] = np.sqrt(S[:r])[:, np.newaxis] * Vt[:r, :]

    W_approx = B @ A #3

    return B, A, W_approx

d = 6   #4
W = np.random.randint(0, 10, size=(d, d)) #5

r = 5  # Rank for the decomposition #6

B, A, W_approx = lora_decomposition(W, r) #7

Inspect the executable decomposition that creates B and A, forms W_approx = B @ A, and applies a chosen rank to a sample matrix. section 8.3.1

A library-based setup can configure rank, target modules, scaling, gradient checkpointing, and randomization through Unsloth’s PEFT interface. The demonstrated configuration targets Qwen3-4B-Base with a rank of 32. section 8.3.1

How Transformers in Action covers it

liveBook manning.com

Read Transformers in Action on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

LLMs in ProductionLow-rank adaptation is a parameter-efficient fine-tuning method that adapts a pretrained model through trainable low-rank updates while leaving most original parameters fixed.
liveBook manning.com

Building Reliable AI SystemsLow-Rank Adaptation is a parameter-efficient fine-tuning method that trains added low-rank matrices while keeping a pretrained model's weights frozen.
liveBook manning.com

Introduction to Generative AI, Second EditionLow-rank adaptation is a parameter-efficient fine-tuning technique that trains low-rank update matrices while keeping most pretrained parameters fixed.
liveBook manning.com

Generative AI in ActionLow-rank adaptation is a parameter-efficient technique that adapts a pretrained model by training small low-rank matrices while leaving the original model weights frozen.
liveBook manning.com

Build a Large Language Model (From Scratch)A parameter-efficient fine-tuning method that adapts a model by training low-rank updates instead of all weights.
liveBook manning.com

The Complete Obsolete Guide to Generative AILoRA is a parameter-efficient fine-tuning technique that adapts a pretrained model by learning low-rank updates rather than modifying all model parameters.
liveBook manning.com

Deep Learning with R, Third EditionLow-rank adaptation is a parameter-efficient fine-tuning technique that freezes a model's original weights and trains a low-rank update to selected projections.
liveBook manning.com

The AI Pocket BookA fine-tuning method that adds small trainable layers or low-rank updates while leaving the original model largely intact.

Compare how 9 books explain low-rank adaptation →

Related concepts

parameter-efficient fine-tuningkind of · Parameter-efficient fine-tuning adapts a pretrained model by training a limited subset of its parameters or added components rather than all its parameters.
QLoRAused by · QLoRA is a parameter-efficient fine-tuning method that quantizes a pretrained model’s weights while training low-rank adapters to reduce memory requirements.
low-rank approximationuses · A low-rank approximation represents a matrix by a lower-rank matrix or factorization, often retaining important structure while reducing storage or computation.

This explainer was generated by Manning's knowledge graph from the text of Transformers in Action. The section links point to the parts of the book it draws on.