QLoRA reduces fine-tuning memory requirements by combining a quantized, frozen base model with trainable low-rank adapters. It is useful when a model does not fit in GPU memory for training at higher precision. If it does fit, training without quantization is faster. section 7.2 section 7.4
NF4 stores weights in four bits, concentrating representable values near zero, where pretrained model weights are densest. This preserves their distribution more faithfully than uniform 4-bit quantization, but still loses precision. NF4 is a storage format: weights are dequantized to 16-bit precision for matrix operations. section 7.2
Figure 7.3 Weight distribution after quantization and dequantization across three methods. 8-bit uniform (left) closely matches the original Gaussian. 4-bit uniform (center) introduces severe distortion, with values collapsing into a small number of discrete levels. NF4 (right), also 4-bit, recovers the distribution more faithfully by concentrating its quantization levels where weights are densest.
Compare the weight distributions produced by 8-bit uniform, 4-bit uniform, and NF4 quantization. section 7.2
A clinical extraction workflow uses QLoRA to replace a long schema instruction with a minimal prompt:
- Format each training example as system, user, and assistant messages, with the clinical note as user content and the JSON label as assistant content. section 7.3.3
- Prepare the quantized model for k-bit training, then attach LoRA adapters with PEFT. The configuration uses rank 8, alpha 16, dropout 0.05, and use_dora=False; it targets attention and MLP projections. section 7.3.3
- Train with TRL's SFTTrainer: 360 samples, batch size 8, and three epochs produce 135 steps. The L4 configuration uses a 2e-4 learning rate, warmup, cosine scheduling, and BF16. section 7.3.3
- Check the trainable parameter count and evaluate both schema compliance and general capabilities against the base model. section 7.3.3 section 7.3.5
Listing 7.10 LoRA configuration
lora_config = LoraConfig(
r=8, #1
lora_alpha=16, #2
lora_dropout=0.05, #3
bias="none",
task_type="CAUSAL_LM",
use_dora=False, #4
target_modules=[
"q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj",
], #5
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
Inspect the adapter configuration and the attention and MLP projection names. section 7.3.3
Listing 7.11 Configuring and training QLoRA
from trl import SFTTrainer, SFTConfig #1
training_args = SFTConfig(
output_dir="./qlora_output",
num_train_epochs=3, #2
per_device_train_batch_size=8,
gradient_accumulation_steps=1,
learning_rate=2e-4, #3
lr_scheduler_type="cosine", #4
warmup_steps=7, #5
fp16=False,
bf16=True, #6
logging_steps=10,
save_strategy="no",
max_length=MAX_SEQ_LENGTH,
dataset_text_field="text", #7
report_to="none",
)
trainer = SFTTrainer(
model=model,
train_dataset=train_dataset,
args=training_args,
)
trainer.train()
Inspect the training settings and SFTTrainer invocation. section 7.3.3
Clinical JSON extraction
SmolLM2-1.7B-Instruct is trained to extract clinical information into a strict JSON schema. The adapter configuration trains approximately 0.53% of parameters. On the 40 test cases, schema compliance rises from 85% with the base model's long prompt to 100% with the QLoRA model's “Extract:” prompt. General-capability benchmarks show a trade-off, including a reported 10.3% drop on ARC-Easy. section 7.3 section 7.3.3 section 7.3.5
QLoRA trained faster than QDoRA in this experiment because it did not manage separate magnitude vectors. The quality results did not establish a clear winner. To test other configurations, vary the training-set size or target modules while keeping the test set and other settings comparable. section 7.3.5 section 7.5






