liveBook · Manning

QLoRA

As explained in Rearchitecting LLMs

QLoRA is a parameter-efficient fine-tuning technique that quantizes a base model and trains low-rank adapters while preserving the quantized model weights.

QLoRA reduces fine-tuning memory requirements by combining a quantized, frozen base model with trainable low-rank adapters. It is useful when a model does not fit in GPU memory for training at higher precision. If it does fit, training without quantization is faster. section 7.2 section 7.4

NF4 stores weights in four bits, concentrating representable values near zero, where pretrained model weights are densest. This preserves their distribution more faithfully than uniform 4-bit quantization, but still loses precision. NF4 is a storage format: weights are dequantized to 16-bit precision for matrix operations. section 7.2

Figure 7.3 Weight distribution after quantization and dequantization across three methods. 8-bit uniform (left) closely matches the original Gaussian. 4-bit uniform (center) introduces severe distortion, with values collapsing into a small number of discrete levels. NF4 (right), also 4-bit, recovers the distribution more faithfully by concentrating its quantization levels where weights are densest.

Compare the weight distributions produced by 8-bit uniform, 4-bit uniform, and NF4 quantization. section 7.2

A clinical extraction workflow uses QLoRA to replace a long schema instruction with a minimal prompt:

  • Format each training example as system, user, and assistant messages, with the clinical note as user content and the JSON label as assistant content. section 7.3.3
  • Prepare the quantized model for k-bit training, then attach LoRA adapters with PEFT. The configuration uses rank 8, alpha 16, dropout 0.05, and use_dora=False; it targets attention and MLP projections. section 7.3.3
  • Train with TRL's SFTTrainer: 360 samples, batch size 8, and three epochs produce 135 steps. The L4 configuration uses a 2e-4 learning rate, warmup, cosine scheduling, and BF16. section 7.3.3
  • Check the trainable parameter count and evaluate both schema compliance and general capabilities against the base model. section 7.3.3 section 7.3.5
Listing 7.10 LoRA configuration
lora_config = LoraConfig(
    r=8,    #1               
    lora_alpha=16, #2         
    lora_dropout=0.05, #3
    bias="none",
    task_type="CAUSAL_LM",
    use_dora=False, #4                 
    target_modules=[                
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ], #5
)
 
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()

Inspect the adapter configuration and the attention and MLP projection names. section 7.3.3

Listing 7.11 Configuring and training QLoRA
from trl import SFTTrainer, SFTConfig #1
 
training_args = SFTConfig( 
    output_dir="./qlora_output",
    num_train_epochs=3, #2
    per_device_train_batch_size=8,
    gradient_accumulation_steps=1,
    learning_rate=2e-4, #3
    lr_scheduler_type="cosine", #4
    warmup_steps=7, #5
    fp16=False,
    bf16=True, #6
    logging_steps=10,
    save_strategy="no",
    max_length=MAX_SEQ_LENGTH,         
    dataset_text_field="text", #7
    report_to="none",
)
 
trainer = SFTTrainer(
    model=model,
    train_dataset=train_dataset,
    args=training_args,
)
trainer.train()
 

Inspect the training settings and SFTTrainer invocation. section 7.3.3

Clinical JSON extraction

SmolLM2-1.7B-Instruct is trained to extract clinical information into a strict JSON schema. The adapter configuration trains approximately 0.53% of parameters. On the 40 test cases, schema compliance rises from 85% with the base model's long prompt to 100% with the QLoRA model's “Extract:” prompt. General-capability benchmarks show a trade-off, including a reported 10.3% drop on ARC-Easy. section 7.3 section 7.3.3 section 7.3.5

QLoRA trained faster than QDoRA in this experiment because it did not manage separate magnitude vectors. The quality results did not establish a clear winner. To test other configurations, vary the training-set size or target modules while keeping the test set and other settings comparable. section 7.3.5 section 7.5

How Rearchitecting LLMs covers it

How deep it goes
Explains NF4 and low-rank adaptation, then configures training, parameter checks, and evaluation against a baseline.
Examples
Uses clinical-note extraction into JSON and exercises varying training-set size and attention or MLP target modules.
What it assumes
Introduces low-rank decomposition and quantization before combining them in a training workflow.
What you can do after
Apply QLoRA via PEFT and TRL to specialize a quantized model and evaluate schema compliance and capability retention.
How it connects
Connects low-rank adaptation, NF4 quantization, supervised fine-tuning, and DoRA.

Also explained in

Compare how 7 books explain QLoRA →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.