liveBook · Manning

QLoRA

As explained in LLM Customization and Fine-Tuning

A LoRA-based fine-tuning method that loads the base model in low-bit precision to reduce memory use during adaptation.

When a model is too large to fine-tune within the available GPU memory, QLoRA offers a lower-memory alternative to LoRA. It loads the frozen base model in 4-bit precision and trains small, higher-precision adapter matrices instead of updating the base weights. The tradeoff is slower training and a possible accuracy cost from quantization. section 5.1.6 section 5.5

QLoRA typically uses NormalFloat4 (NF4), a 4-bit format designed for normally distributed weights, and double quantization, which compresses quantization scale and offset values. The base weights remain frozen; only the adapter parameters are trained. section 5.5

Listing 5.5 Loading a base model with 4-bit quantization for QLoRA
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, prepare_model_for_kbit_training
from trl import SFTTrainer
 
# Configure 4-bit quantization
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",             # NormalFloat4 quantization
    bnb_4bit_use_double_quant=True,  # double quantization, more savings
    bnb_4bit_compute_dtype=torch.bfloat16  # Compute in bfloat16 for speed
)
 
# Load model in 4-bit
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,
    device_map="auto"
)
 
# k-bit training prep: gradient checkpointing on 4-bit weights
model = prepare_model_for_kbit_training(model)
 
# Rest is the same as step 2: LoRA config, load data, train with SFTTrainer
lora_config = LoraConfig(r=8, lora_alpha=16, ...)
trainer = SFTTrainer(model=model, peft_config=lora_config, ...)
trainer.train()

The loading configuration uses bitsandbytes 4-bit NF4 and double quantization, prepares the model for k-bit training, and then applies the usual LoRA training pipeline. section 5.1.6

Qwen3-4B under a memory constraint

For Qwen3-4B, QLoRA requires approximately 4–6 GB of GPU memory, compared with 8–12 GB for regular LoRA. On the same 24 GB GPU, a run with 450 examples over three epochs took about 14–15 minutes with QLoRA versus 11 minutes with LoRA. The QLoRA configuration uses rank 8 and alpha 16 by default; training data, evaluation, and inference workflows otherwise remain the same. section 5.1.6

Table 5.5 Typical memory requirements for fine-tuning a model of various sizes
Model Size
Full Fine-Tuning (VRAM)
LoRA (VRAM)
QLoRA (VRAM)
Notes
7 B
~60 to 67 GB
16 to 20 GB
8 to 10 GB
Fits on a single 24 GB GPU using LoRA. QLoRA enables larger batch sizes or longer contexts.
13 B
~120 to 130 GB
30 to 36 GB
10 to 18 GB
LoRA often requires two 24 GB GPUs for 13B; QLoRA may fit on one high-end consumer GPU.
65 B
600 GB+ (multi-GPU)
~146 GB
~48 GB+
65B LoRA needs a multi-GPU node (e.g., 8x H100). QLoRA fits 65B on a single 48 GB GPU, the QLoRA paper’s headline result [2]; an 80 GB H100 or H200 gives headroom for longer contexts.

Compare typical VRAM requirements for full fine-tuning, LoRA, and QLoRA across 7B, 13B, and 65B models; actual requirements depend on the run. section 5.5

Use QLoRA when the memory savings justify the tradeoffs:

  • Start with LoRA when it fits the hardware; consider QLoRA when a larger model or tighter memory budget makes regular LoRA impractical. section 5.5 section 5.1.6
  • Consider retrieval-augmented generation for frequently changing knowledge or answers requiring verifiable citations. Adaptation is not a substitute for a retrieval pipeline in those cases. section 1.6.2
  • Evaluate task quality and test for safety regressions after tuning. Include explicit refusal examples in the training data. section 5.13
  • Plan serving separately: a common path is to merge the adapter into the full-precision base and then quantize the merged model. GPTQ or AWQ typically offer higher inference throughput than the bitsandbytes training format. section 5.5

How LLM Customization and Fine-Tuning covers it

How deep it goes
Explains 4-bit loading, NF4, double quantization, and training configuration, while deferring deployment quantization.
Examples
Uses a Qwen3-4B training run and compares VRAM requirements for 7B, 13B, and 65B models.
What you can do after
Apply it via bitsandbytes, Hugging Face PEFT, and SFTTrainer to train a quantized-base LoRA adapter.
How it connects
Connects LoRA and quantization to full fine-tuning and retrieval-augmented generation as adaptation choices.

Also explained in

Compare how 7 books explain QLoRA →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.