DoRA, or Weight-Decomposed LoRA, adapts pretrained models by separating each weight matrix’s magnitude from its direction. It targets specialized behavior while preserving more of the model’s foundational capabilities than an undifferentiated update. section 3.5.3
DoRA’s implementation follows this structure:
- Decompose each pretrained matrix W₀ into a magnitude vector m and normalized directional matrix V₀. section 3.5.3
- Apply LoRA’s low-rank matrices A and B to the directional component instead of updating the entire weight matrix directly. section 3.5.3
- Train m, A, and B while retaining the pretrained directional component, reducing the number of parameters requiring adaptation. section 3.5.3
- Use DoRA when specialization requires finer control than standard LoRA, while weighing adaptation depth, compute cost, and preservation of pretrained knowledge. section 3.8
Medical-record adaptation
A general-purpose model adapted to medical records can learn medical terminology through directional updates while retaining coherent sentence construction associated with its pretrained capabilities. section 3.5.3
Figure 3.13 DoRA architecture. DoRA improves on standard LoRA by decomposing the pre-trained weight into magnitude and direction components for fine-tuning. LoRA is then applied to efficiently update the direction component. Note that || ||c denotes the vector-wise norm of a matrix across each column vector.

The architecture figure shows the pretrained weight split into magnitude and direction, with LoRA updating the directional component. section 3.5.3


