liveBook · Manning

DoRA

As explained in Building Specialized AI Systems

DoRA is a parameter-efficient fine-tuning method that decomposes pretrained weights into magnitude and direction components and applies low-rank adaptation to the directional component.

DoRA, or Weight-Decomposed LoRA, adapts pretrained models by separating each weight matrix’s magnitude from its direction. It targets specialized behavior while preserving more of the model’s foundational capabilities than an undifferentiated update. section 3.5.3

DoRA’s implementation follows this structure:

  • Decompose each pretrained matrix W₀ into a magnitude vector m and normalized directional matrix V₀. section 3.5.3
  • Apply LoRA’s low-rank matrices A and B to the directional component instead of updating the entire weight matrix directly. section 3.5.3
  • Train m, A, and B while retaining the pretrained directional component, reducing the number of parameters requiring adaptation. section 3.5.3
  • Use DoRA when specialization requires finer control than standard LoRA, while weighing adaptation depth, compute cost, and preservation of pretrained knowledge. section 3.8

Medical-record adaptation

A general-purpose model adapted to medical records can learn medical terminology through directional updates while retaining coherent sentence construction associated with its pretrained capabilities. section 3.5.3

Figure 3.13 DoRA architecture. DoRA improves on standard LoRA by decomposing the pre-trained weight into magnitude and direction components for fine-tuning. LoRA is then applied to efficiently update the direction component. Note that || ||c denotes the vector-wise norm of a matrix across each column vector.

The architecture figure shows the pretrained weight split into magnitude and direction, with LoRA updating the directional component. section 3.5.3

How Building Specialized AI Systems covers it

How deep it goes
Explains matrix decomposition, trainable components, and forward-pass normalization, but does not provide a complete training workflow.
Examples
Uses medical-record analysis and a magnitude-direction architecture figure to illustrate specialized adaptation.
What it assumes
Assumes pretrained weights and LoRA-style low-rank updates, while defining magnitude and directional components explicitly.
What you can do after
Apply DoRA via PEFT’s LoraConfig by enabling the use_dora option.
How it connects
Connects LoRA, parameter-efficient fine-tuning, pretrained weights, and full fine-tuning through adaptation and preservation trade-offs.

Also explained in

Compare how 3 books explain DoRA →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Building Specialized AI Systems. The section links point to the parts of the book it draws on.