DoRA
DoRA is a parameter-efficient fine-tuning method that decomposes model weights into separately tunable magnitude and directional components, typically applying low-rank adaptation to the directional component.
Rearchitecting LLMs
DoRA is a parameter-efficient adaptation method that decomposes model weights into separate magnitude and direction components and updates them independently.
- How it is used
- Specializes model behavior efficiently while preserving a frozen base model.
- How deep it goes
- Explains decomposition, normalization, and parameter counts with a simulated PyTorch update, but leaves rank selection empirical.
- Examples
- Uses a 2,048-dimensional weight example and compares schema compliance under an aggressive rank-1 configuration.
- What you can do after
- Evaluate DoRA by comparing task performance and general capabilities against a baseline, including on a semantic adaptation task.
Transformers in Action
DoRA is a parameter-efficient fine-tuning method that decomposes model weights into separately tunable magnitude and directional components.
- How it is used
- Deploys magnitude-direction reparameterization to adapt pretrained large language models with limited parameter updates.
- How deep it goes
- Explains decomposition, column-wise normalization, initialization, frozen and trainable components, and low-rank directional updates.
- Examples
- Uses an architectural overview showing magnitude-direction separation and LoRA refinement of the directional component.
- What you can do after
- Apply DoRA when faster convergence or interpretable magnitude-direction adaptation is more important than minimum memory usage.
Building Specialized AI Systems
DoRA is a parameter-efficient fine-tuning method that decomposes pretrained weights into magnitude and direction components and applies low-rank adaptation to the directional component.
- How it is used
- Supports parameter-efficient specialization when preserving foundational pretrained behavior matters.
- How deep it goes
- Explains matrix decomposition, trainable components, and forward-pass normalization, but does not provide a complete training workflow.
- Examples
- Uses medical-record analysis and a magnitude-direction architecture figure to illustrate specialized adaptation.
- What you can do after
- Apply DoRA via PEFT’s LoraConfig by enabling the use_dora option.
What to know first
- fine-tuningFine-tuning is the continued training of a pretrained model on targeted data or objectives to adapt its parameters or behavior to a task, domain, or use.
- low-rank adaptationLow-rank adaptation (LoRA) is a parameter-efficient fine-tuning technique that adapts a model by learning low-rank updates to its weights, typically while keeping most original weights fixed.
- low-rank approximationA low-rank approximation represents a matrix or transformation with a lower-rank matrix or factorization, often retaining dominant structure while reducing storage or computation.
- parameterA parameter is a value or named input that specifies or influences the behavior, computation, or output of a function, model, or system.
Related concepts
- low-rank adaptationkind of · Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning technique that adapts a model by learning low-rank updates to its weights, typically while keeping most original weights fixed.
- parameter-efficient fine-tuningkind of · Parameter-efficient fine-tuning adapts a pretrained model by training a limited set of existing or added parameters while leaving most of the model unchanged.
- low-rank approximationuses · A low-rank approximation represents a matrix or transformation with a lower-rank matrix or factorization, often retaining dominant structure while reducing storage or computation.
Questions about DoRA
What is DoRA?
DoRA is a parameter-efficient fine-tuning method that decomposes model weights into separately tunable magnitude and directional components, typically applying low-rank adaptation to the directional component.
Which Manning books explain DoRA?
Rearchitecting LLMs, Transformers in Action and Building Specialized AI Systems.
What should I know before learning DoRA?
Manning books that teach DoRA usually introduce fine-tuning, low-rank adaptation, low-rank approximation and parameter first.
What is the difference between DoRA and weight normalization?
DoRA is a parameter-efficient fine-tuning method that decomposes model weights into separately tunable magnitude and directional components, typically applying low-rank adaptation to the directional component. Weight normalization is the rescaling of model weights to satisfy a specified normalization condition, often by separating each vector’s magnitude from its direction.
What is DoRA a kind of?
DoRA is a kind of low-rank adaptation. Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning technique that adapts a model by learning low-rank updates to its weights, typically while keeping most original weights fixed.
What are the parts of DoRA?
DoRA includes weight decomposition and directional component.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


