QDoRA combines quantization with weight-decomposition low-rank adaptation to specialize a model without training all its parameters. It separates weight magnitude from direction, adding trainable magnitude vectors to the low-rank updates used for direction. This gives magnitude and direction more independence than LoRA, at the cost of additional parameters and training time. section 7.3 section 7.4 section 7.3.5
With NF4, the base weights use a 4-bit storage format, but matrix multiplication is performed after on-the-fly dequantization to 16-bit precision. In one T4 training setup, QDoRA peaked at 2.69 GB, versus 2.34 GB for QLoRA; these figures include the base model, adapters, gradients, and optimizer states. section 7.2
To switch a QLoRA training workflow to QDoRA:
- Clear the existing model and reload a clean base model with the quantization configuration. Prepare it for k-bit training. section 7.3.4
- Set use_dora=True in LoraConfig, attach the adapter, and inspect the trainable parameter count. The example uses rank 8 and targets attention and MLP projections. section 7.3.4
- Train with the same SFTConfig and SFTTrainer settings as the QLoRA comparison, then evaluate both task compliance and general capabilities. section 7.3.4
Listing 7.13 QDoRA configuration
dora_config = LoraConfig(
r=8,
lora_alpha=16,
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
use_dora=True, #A
target_modules=[
"q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj",
],
)
model = get_peft_model(model, dora_config)
model.print_trainable_parameters()
The configuration shows the DoRA switch, adapter settings, and target modules used in the comparison. section 7.3.4
Clinical JSON extraction
SmolLM2-1.7B-Instruct was specialized to extract clinical information from notes as schema-constrained JSON. QDoRA trained 9,682,944 parameters, about 639,000 more than QLoRA. Both reached 100% schema compliance with a minimal prompt, compared with 85% for the baseline using a strict prompt. Both also showed moderate general-benchmark drops, and QDoRA did not measurably improve on QLoRA in this experiment. section 7.3 section 7.3.4 section 7.3.5
For this format-adaptation task, QLoRA is the more efficient choice because its quality was comparable and it trained faster. A possible QDoRA advantage on tasks demanding deeper semantic adaptation remains a hypothesis to test, not a conclusion from the clinical experiment. section 7.3.5 section 7.5
