liveBook · Manning

QDoRA

As explained in Rearchitecting LLMs

QDoRA is a quantized weight-decomposition low-rank adaptation technique that adds magnitude parameters to direction-based low-rank updates.

QDoRA combines quantization with weight-decomposition low-rank adaptation to specialize a model without training all its parameters. It separates weight magnitude from direction, adding trainable magnitude vectors to the low-rank updates used for direction. This gives magnitude and direction more independence than LoRA, at the cost of additional parameters and training time. section 7.3 section 7.4 section 7.3.5

With NF4, the base weights use a 4-bit storage format, but matrix multiplication is performed after on-the-fly dequantization to 16-bit precision. In one T4 training setup, QDoRA peaked at 2.69 GB, versus 2.34 GB for QLoRA; these figures include the base model, adapters, gradients, and optimizer states. section 7.2

To switch a QLoRA training workflow to QDoRA:

  • Clear the existing model and reload a clean base model with the quantization configuration. Prepare it for k-bit training. section 7.3.4
  • Set use_dora=True in LoraConfig, attach the adapter, and inspect the trainable parameter count. The example uses rank 8 and targets attention and MLP projections. section 7.3.4
  • Train with the same SFTConfig and SFTTrainer settings as the QLoRA comparison, then evaluate both task compliance and general capabilities. section 7.3.4
Listing 7.13 QDoRA configuration
dora_config = LoraConfig(
    r=8,
    lora_alpha=16,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    use_dora=True, #A
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
)
 
model = get_peft_model(model, dora_config)
model.print_trainable_parameters()

The configuration shows the DoRA switch, adapter settings, and target modules used in the comparison. section 7.3.4

Clinical JSON extraction

SmolLM2-1.7B-Instruct was specialized to extract clinical information from notes as schema-constrained JSON. QDoRA trained 9,682,944 parameters, about 639,000 more than QLoRA. Both reached 100% schema compliance with a minimal prompt, compared with 85% for the baseline using a strict prompt. Both also showed moderate general-benchmark drops, and QDoRA did not measurably improve on QLoRA in this experiment. section 7.3 section 7.3.4 section 7.3.5

For this format-adaptation task, QLoRA is the more efficient choice because its quality was comparable and it trained faster. A possible QDoRA advantage on tasks demanding deeper semantic adaptation remains a hypothesis to test, not a conclusion from the clinical experiment. section 7.3.5 section 7.5

How Rearchitecting LLMs covers it

How deep it goes
Explains magnitude-direction decomposition and walks through configuration, training, and comparative evaluation.
Examples
Uses clinical-note extraction into strict JSON and proposes testing semantic adaptation on a databricks-dolly-15k subset.
What you can do after
Applies QDoRA via PEFT configuration and compares schema compliance, general capabilities, and training cost against QLoRA.
How it connects
Connects QDoRA to LoRA, DoRA, QLoRA, NF4 quantization, and supervised fine-tuning.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.