liveBook · Manning

DoRA

As explained in Rearchitecting LLMs

DoRA is a parameter-efficient adaptation method that decomposes model weights into separate magnitude and direction components and updates them independently.

DoRA (Weight-Decomposed Low-Rank Adaptation) specializes a model while keeping its base weights frozen. It extends LoRA by separating each weight matrix into magnitude and direction, then learning a low-rank directional update alongside independent magnitude parameters. This gives adaptation more flexibility without full fine-tuning. section 7.1 section 7.1.2

A LoRA update changes magnitude and direction together. DoRA instead uses LoRA matrices to adjust direction while training magnitude separately. Reported magnitude-direction correlations are -0.62 for full fine-tuning, +0.83 for LoRA, and -0.31 for DoRA; these figures motivate the decomposition rather than guarantee an outcome for every task. section 7.1.2 section 7.4

A simplified DoRA update proceeds in four stages:

  • Compute a magnitude vector m from the column norms of the frozen weight matrix W0, and set the initial direction V = W0 / m. section 7.1.2
  • Form a low-rank directional update from matrices A and B, then add it to V. section 7.1.2
  • Normalize the updated direction by its column norms, then scale it by m to reconstruct the adapted weight. section 7.1.2
  • Train the low-rank matrices and magnitude parameters while leaving the original weight frozen. section 7.1 section 7.1.2

Inspecting a directional update

For a 2,048 × 2,048 weight and rank 4, A and B contain 16,384 parameters; the magnitude vector adds 2,048. The adapted weight retains the original shape. The demonstration uses a random weight and a simulated trained B, rather than running a training loop. Rank 4 illustrates the mechanism; rank selection remains empirical. section 7.1.2

Listing 7.3 DoRA applied to q_proj of SmolLM2-1.7B
import torch
 
d = 2048
r = 4 #1
 
W0 = torch.randn(d, d)  #2
 
m = torch.norm(W0, dim=0, keepdim=True)  #3
V = W0 / m #4                                   
 
A = torch.randn(d, r)
B = torch.zeros(r, d)
 
B_trained = torch.randn(r, d) * 0.01 #5
 
V_updated     = V + A @ B_trained #6
V_updated_norm = torch.norm(V_updated, dim=0, keepdim=True)
W_dora = m * (V_updated / V_updated_norm) #7         
 
params_lora   = d * r + r * d
params_dora_m = d                                
 
print(f"W0 shape:                    {W0.shape}")
print(f"W_dora shape:                {W_dora.shape}")
print()
print(f"Parameters in A + B:         {params_lora:,}")
print(f"Additional magnitude params: {params_dora_m:,}")
print()
print(f"Sample W0[0, :4]:     {W0[0, :4]}")
print(f"Sample W_dora[0, :4]: {W_dora[0, :4]}")

Inspect the PyTorch decomposition, simulated directional update, normalization, and parameter counts. section 7.1.2

DoRA is intended to preserve general capabilities during semantic adaptation, but its advantage may be marginal for purely structural tasks. Establish a general-capabilities baseline and compare task performance and forgetting after training. In one aggressive configuration—rank 1, one epoch, and a larger learning rate—both LoRA and DoRA retained 100% schema compliance with better capability preservation. To test a semantic advantage, compare techniques on a reasoning-oriented dataset while keeping the remaining configuration fixed. section 7.6 section 7.5

For memory-constrained training, quantization can reduce the base model's GPU footprint. After training, an adapter can be merged into the base model or kept separate for dynamic loading of different specializations. section 7.0 section 7.6

How Rearchitecting LLMs covers it

How deep it goes
Explains decomposition, normalization, and parameter counts with a simulated PyTorch update, but leaves rank selection empirical.
Examples
Uses a 2,048-dimensional weight example and compares schema compliance under an aggressive rank-1 configuration.
What you can do after
Evaluate DoRA by comparing task performance and general capabilities against a baseline, including on a semantic adaptation task.
How it connects
Connects low-rank adaptation to full fine-tuning, catastrophic forgetting, quantization, and adapter deployment.

Also explained in

Compare how 3 books explain DoRA →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.