When a model is too large to fine-tune within the available GPU memory, QLoRA offers a lower-memory alternative to LoRA. It loads the frozen base model in 4-bit precision and trains small, higher-precision adapter matrices instead of updating the base weights. The tradeoff is slower training and a possible accuracy cost from quantization. section 5.1.6 section 5.5
QLoRA typically uses NormalFloat4 (NF4), a 4-bit format designed for normally distributed weights, and double quantization, which compresses quantization scale and offset values. The base weights remain frozen; only the adapter parameters are trained. section 5.5
Listing 5.5 Loading a base model with 4-bit quantization for QLoRA
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, prepare_model_for_kbit_training
from trl import SFTTrainer
# Configure 4-bit quantization
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4", # NormalFloat4 quantization
bnb_4bit_use_double_quant=True, # double quantization, more savings
bnb_4bit_compute_dtype=torch.bfloat16 # Compute in bfloat16 for speed
)
# Load model in 4-bit
model = AutoModelForCausalLM.from_pretrained(
model_name,
quantization_config=bnb_config,
device_map="auto"
)
# k-bit training prep: gradient checkpointing on 4-bit weights
model = prepare_model_for_kbit_training(model)
# Rest is the same as step 2: LoRA config, load data, train with SFTTrainer
lora_config = LoraConfig(r=8, lora_alpha=16, ...)
trainer = SFTTrainer(model=model, peft_config=lora_config, ...)
trainer.train()The loading configuration uses bitsandbytes 4-bit NF4 and double quantization, prepares the model for k-bit training, and then applies the usual LoRA training pipeline. section 5.1.6
Qwen3-4B under a memory constraint
For Qwen3-4B, QLoRA requires approximately 4–6 GB of GPU memory, compared with 8–12 GB for regular LoRA. On the same 24 GB GPU, a run with 450 examples over three epochs took about 14–15 minutes with QLoRA versus 11 minutes with LoRA. The QLoRA configuration uses rank 8 and alpha 16 by default; training data, evaluation, and inference workflows otherwise remain the same. section 5.1.6
Table 5.5 Typical memory requirements for fine-tuning a model of various sizes
|
Model Size
|
Full
Fine-Tuning (VRAM)
|
LoRA (VRAM)
|
QLoRA (VRAM)
|
Notes
|
|
7 B
|
~60 to 67 GB
|
16 to 20 GB
|
8 to 10 GB
|
Fits on a
single 24 GB GPU using LoRA. QLoRA enables larger batch sizes or longer
contexts.
|
|
13 B
|
~120 to 130 GB
|
30 to 36 GB
|
10 to 18 GB
|
LoRA often
requires two 24 GB GPUs for 13B; QLoRA may fit on one high-end consumer GPU.
|
|
65 B
|
600 GB+
(multi-GPU)
|
~146 GB
|
~48 GB+
|
65B LoRA needs
a multi-GPU node (e.g., 8x H100). QLoRA fits 65B on a single 48 GB GPU, the
QLoRA paper’s headline result [2]; an 80 GB H100 or H200 gives headroom for
longer contexts.
|
Compare typical VRAM requirements for full fine-tuning, LoRA, and QLoRA across 7B, 13B, and 65B models; actual requirements depend on the run. section 5.5
Use QLoRA when the memory savings justify the tradeoffs:
- Start with LoRA when it fits the hardware; consider QLoRA when a larger model or tighter memory budget makes regular LoRA impractical. section 5.5 section 5.1.6
- Consider retrieval-augmented generation for frequently changing knowledge or answers requiring verifiable citations. Adaptation is not a substitute for a retrieval pipeline in those cases. section 1.6.2
- Evaluate task quality and test for safety regressions after tuning. Include explicit refusal examples in the training data. section 5.13
- Plan serving separately: a common path is to merge the adapter into the full-precision base and then quantize the merged model. GPTQ or AWQ typically offer higher inference throughput than the bitsandbytes training format. section 5.5






