DoRA (Weight-Decomposed Low-Rank Adaptation) specializes a model while keeping its base weights frozen. It extends LoRA by separating each weight matrix into magnitude and direction, then learning a low-rank directional update alongside independent magnitude parameters. This gives adaptation more flexibility without full fine-tuning. section 7.1 section 7.1.2
A LoRA update changes magnitude and direction together. DoRA instead uses LoRA matrices to adjust direction while training magnitude separately. Reported magnitude-direction correlations are -0.62 for full fine-tuning, +0.83 for LoRA, and -0.31 for DoRA; these figures motivate the decomposition rather than guarantee an outcome for every task. section 7.1.2 section 7.4
A simplified DoRA update proceeds in four stages:
- Compute a magnitude vector m from the column norms of the frozen weight matrix W0, and set the initial direction V = W0 / m. section 7.1.2
- Form a low-rank directional update from matrices A and B, then add it to V. section 7.1.2
- Normalize the updated direction by its column norms, then scale it by m to reconstruct the adapted weight. section 7.1.2
- Train the low-rank matrices and magnitude parameters while leaving the original weight frozen. section 7.1 section 7.1.2
Inspecting a directional update
For a 2,048 × 2,048 weight and rank 4, A and B contain 16,384 parameters; the magnitude vector adds 2,048. The adapted weight retains the original shape. The demonstration uses a random weight and a simulated trained B, rather than running a training loop. Rank 4 illustrates the mechanism; rank selection remains empirical. section 7.1.2
Listing 7.3 DoRA applied to q_proj of SmolLM2-1.7B
import torch
d = 2048
r = 4 #1
W0 = torch.randn(d, d) #2
m = torch.norm(W0, dim=0, keepdim=True) #3
V = W0 / m #4
A = torch.randn(d, r)
B = torch.zeros(r, d)
B_trained = torch.randn(r, d) * 0.01 #5
V_updated = V + A @ B_trained #6
V_updated_norm = torch.norm(V_updated, dim=0, keepdim=True)
W_dora = m * (V_updated / V_updated_norm) #7
params_lora = d * r + r * d
params_dora_m = d
print(f"W0 shape: {W0.shape}")
print(f"W_dora shape: {W_dora.shape}")
print()
print(f"Parameters in A + B: {params_lora:,}")
print(f"Additional magnitude params: {params_dora_m:,}")
print()
print(f"Sample W0[0, :4]: {W0[0, :4]}")
print(f"Sample W_dora[0, :4]: {W_dora[0, :4]}")
Inspect the PyTorch decomposition, simulated directional update, normalization, and parameter counts. section 7.1.2
DoRA is intended to preserve general capabilities during semantic adaptation, but its advantage may be marginal for purely structural tasks. Establish a general-capabilities baseline and compare task performance and forgetting after training. In one aggressive configuration—rank 1, one epoch, and a larger learning rate—both LoRA and DoRA retained 100% schema compliance with better capability preservation. To test a semantic advantage, compare techniques on a reasoning-oriented dataset while keeping the remaining configuration fixed. section 7.6 section 7.5
For memory-constrained training, quantization can reduce the base model's GPU footprint. After training, an adapter can be merged into the base model or kept separate for dynamic loading of different specializations. section 7.0 section 7.6


