DoRA

As explained in Transformers in Action

DoRA is a parameter-efficient fine-tuning method that decomposes model weights into separately tunable magnitude and directional components.

DoRA is a parameter-efficient fine-tuning method for pretrained large language models. It extends LoRA by separating weight updates into magnitude and direction, reducing parameter changes while preserving pretrained knowledge and lowering training cost. section 8.3

DoRA reparameterizes each pretrained weight matrix into a trainable magnitude vector and normalized directional matrix. The directional update uses LoRA-style low-rank matrices, while the pretrained directional component remains frozen and the magnitude is trainable. Initialization preserves the original model output before adaptation. section 8.3.2

Figure 8.13 An overview of DoRA, which segments the pretrained weight into separate magnitude and direction elements for fine-tuning purposes, utilizing LoRA to effectively refine the direction aspect. It’s important to note that ||.||c represents the vector-wise norm across each column vector in a matrix. Image is taken from [2].

Inspect how DoRA separates pretrained weights into magnitude and direction while applying low-rank adaptation to the directional component. section 8.3.2

Use DoRA when adaptation behavior or convergence matters more than minimizing memory alone:

• Choose DoRA over basic LoRA when decoupling magnitude and direction may provide faster or more nuanced convergence. section 8.3.7
• Prefer quantized methods such as QLoRA or QA-LoRA when memory efficiency is the primary constraint. section 8.3.7
• Relate DoRA to weight normalization carefully: DoRA starts from pretrained weights, avoiding the initialization sensitivity associated with training components from scratch. section 8.3.2

How Transformers in Action covers it

How deep it goes

Explains decomposition, column-wise normalization, initialization, frozen and trainable components, and low-rank directional updates.

Examples

Uses an architectural overview showing magnitude-direction separation and LoRA refinement of the directional component.

What it assumes

Assumes pretrained weights and low-rank adaptation concepts while distinguishing DoRA from weight normalization.

What you can do after

Apply DoRA when faster convergence or interpretable magnitude-direction adaptation is more important than minimum memory usage.

How it connects

Connects LoRA, weight normalization, pretrained weights, quantization, and parameter-efficient fine-tuning.

liveBookmanning.com

Read Transformers in Action on liveBook Get the book at manning.com

Related concepts

• low-rank adaptationkind of · Low-rank adaptation is a parameter-efficient fine-tuning method that adapts a model by learning low-rank updates to its weights, typically while keeping most original weights fixed.
• parameter-efficient fine-tuningkind of · Parameter-efficient fine-tuning adapts a pretrained model by training a limited subset of its parameters or added components rather than all its parameters.
• low-rank approximationuses · A low-rank approximation represents a matrix by a lower-rank matrix or factorization, often retaining important structure while reducing storage or computation.
• weightuses · A weight is a numerical quantity assigned to an element to represent its contribution, influence, importance, strength, or resource amount in a calculation or system.

This explainer was generated by Manning's knowledge graph from the text of Transformers in Action. The section links point to the parts of the book it draws on.