liveBook · Manning

DoRA

Also called weight-decomposed low-rank adaptation

DoRA is a parameter-efficient fine-tuning method that decomposes model weights into separately tunable magnitude and directional components, typically applying low-rank adaptation to the directional component.

How 3 Manning books explain it

Rearchitecting LLMs

DoRA is a parameter-efficient adaptation method that decomposes model weights into separate magnitude and direction components and updates them independently.

How it is used
Specializes model behavior efficiently while preserving a frozen base model.
How deep it goes
Explains decomposition, normalization, and parameter counts with a simulated PyTorch update, but leaves rank selection empirical.
Examples
Uses a 2,048-dimensional weight example and compares schema compliance under an aggressive rank-1 configuration.
What you can do after
Evaluate DoRA by comparing task performance and general capabilities against a baseline, including on a semantic adaptation task.

Read how Rearchitecting LLMs explains DoRA →

Transformers in Action

DoRA is a parameter-efficient fine-tuning method that decomposes model weights into separately tunable magnitude and directional components.

How it is used
Deploys magnitude-direction reparameterization to adapt pretrained large language models with limited parameter updates.
How deep it goes
Explains decomposition, column-wise normalization, initialization, frozen and trainable components, and low-rank directional updates.
Examples
Uses an architectural overview showing magnitude-direction separation and LoRA refinement of the directional component.
What you can do after
Apply DoRA when faster convergence or interpretable magnitude-direction adaptation is more important than minimum memory usage.

Read how Transformers in Action explains DoRA →

Building Specialized AI Systems

DoRA is a parameter-efficient fine-tuning method that decomposes pretrained weights into magnitude and direction components and applies low-rank adaptation to the directional component.

How it is used
Supports parameter-efficient specialization when preserving foundational pretrained behavior matters.
How deep it goes
Explains matrix decomposition, trainable components, and forward-pass normalization, but does not provide a complete training workflow.
Examples
Uses medical-record analysis and a magnitude-direction architecture figure to illustrate specialized adaptation.
What you can do after
Apply DoRA via PEFT’s LoraConfig by enabling the use_dora option.

Read how Building Specialized AI Systems explains DoRA →

What to know first

Related concepts

Questions about DoRA

What is DoRA?

DoRA is a parameter-efficient fine-tuning method that decomposes model weights into separately tunable magnitude and directional components, typically applying low-rank adaptation to the directional component.

Which Manning books explain DoRA?

Rearchitecting LLMs, Transformers in Action and Building Specialized AI Systems.

What should I know before learning DoRA?

Manning books that teach DoRA usually introduce fine-tuning, low-rank adaptation, low-rank approximation and parameter first.

What is the difference between DoRA and weight normalization?

DoRA is a parameter-efficient fine-tuning method that decomposes model weights into separately tunable magnitude and directional components, typically applying low-rank adaptation to the directional component. Weight normalization is the rescaling of model weights to satisfy a specified normalization condition, often by separating each vector’s magnitude from its direction.

What is DoRA a kind of?

DoRA is a kind of low-rank adaptation. Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning technique that adapts a model by learning low-rank updates to its weights, typically while keeping most original weights fixed.

What are the parts of DoRA?

DoRA includes weight decomposition and directional component.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.