liveBook · Manning

Low-rank adaptation

As explained in Rearchitecting LLMs

Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning technique that trains low-rank updates while keeping the original model weights frozen.

Low-rank adaptation (LoRA) specializes a model by training a small set of parameters while keeping its original weights frozen. Compared with full fine-tuning, this reduces training cost and time, but LoRA alone does not address the base model’s memory consumption. section 1.2.3 section 7.4

For a weight matrix W of size d × d, LoRA learns an update ΔW = AB, where A is d × r and B is r × d. The product has the same shape as W and can be added to it. Rank r controls the trade-off: a larger rank provides more expressive capacity but requires more trainable weights. Common starting ranks are 4, 8, and 16. section 7.1.1

Attention projection parameter count

In SmolLM2-1.7B, q_proj is 2048 × 2048. At rank 8, its LoRA matrices contain 2048 × 8 + 8 × 2048 = 32,768 parameters, versus 4,194,304 in the original matrix: a 99.2% reduction for that projection. section 7.1.1

Listing 7.2 LoRA applied to q_proj of SmolLM2-1.7B
import torch
 
d = 2048 #1
r = 8 #2
 
W = torch.randn(d, d) #3
 
A = torch.randn(d, r)
B = torch.zeros(r, d)  #4
 
delta_W = A @ B
 
print(f"W dimensions:        {W.shape}")
print(f"ΔW dimensions:       {delta_W.shape}")
print()
params_W    = d * d
params_lora = d * r + r * d
reduction   = (1 - params_lora / params_W) * 100
print(f"Parameters in W:     {params_W:,}")
print(f"Parameters in A + B: {params_lora:,}")
print(f"Reduction:           {reduction:.1f}%")
 
B_trained = torch.randn(r, d) * 0.01  #5
delta_W_trained = A @ B_trained
 
W_adapted = W + delta_W_trained  #6
 
print(f"\nW dimensions (unchanged):        {W_adapted.shape}")
print(f"Sample weights before adaptation: {W[0, :4]}")
print(f"Sample weights after adaptation:  {W_adapted[0, :4]}")

Check the matrix shapes and parameter reduction in PyTorch; the final update is simulated rather than trained. section 7.1.1

LoRA can follow structural optimization and knowledge recovery: prune the model, recover general capabilities through distillation, then fine-tune on domain data. For example, financial reports can guide pruning and later specialize a recovered model for earnings analysis. section 1.2.5

A function-calling workflow illustrates the implementation choices:

  • Load a post-trained Qwen3-0.6B checkpoint in float16. Disable the cache for gradient checkpointing, match the padding token, and enable input gradients. section 13.3
  • Configure PEFT with rank 16, alpha 32, dropout 0.05, and attention and MLP projection targets. This adapts around 1.5% of the model’s parameters. section 13.3
  • Train for three epochs with a learning rate of 2e-4 and an effective batch size of 8. Compute loss only on completions so training focuses on generating the function call. section 13.3
  • Save the adapter, merge it into the specialist model, restore the cache, disable gradient checkpointing, and switch to evaluation mode. section 13.3
Listing 13.6 LoRA configuration
lora_config = LoraConfig(
    r=16, 
    lora_alpha=32, 
    lora_dropout=0.05,
    bias="none", 
    task_type="CAUSAL_LM", 
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
)
train_model = get_peft_model(train_model, lora_config)
train_model.print_trainable_parameters()

Inspect the PEFT configuration and the exact projection modules selected for adaptation. section 13.3

For tighter memory constraints, QLoRA combines adaptation with 4-bit NormalFloat quantization. DoRA instead separates weight magnitude and direction for training, addressing a difference between LoRA and full fine-tuning. section 7.4

How Rearchitecting LLMs covers it

How deep it goes
Explains matrix shapes and parameter savings, then walks through adapter configuration, training, and merging.
Examples
Calculates SmolLM2 attention-projection savings and configures Qwen3-0.6B for function calling.
What you can do after
Apply it via PEFT to configure, train, save, and merge a function-calling adapter.
How it connects
Connects LoRA with pruning, knowledge distillation, quantization, and weight-decomposed adaptation.

Also explained in

Compare how 17 books explain low-rank adaptation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.