Quantization reduces the precision of model weights to lower memory requirements. Storing weights in 4 bits rather than 16 cuts weight storage by roughly fourfold, with a possible accuracy tradeoff. This can make fine-tuning feasible when a model otherwise exceeds available GPU memory. section 5.5
QLoRA combines 4-bit quantization with LoRA: the quantized base weights remain frozen, while higher-precision adapter weights are trained. Two techniques improve memory efficiency: NormalFloat4 (NF4) allocates more precision near zero for normally distributed weights, and double quantization compresses quantization scale and offset values, saving about 0.4 bits per parameter. section 5.5
Listing 5.5 Loading a base model with 4-bit quantization for QLoRA
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, prepare_model_for_kbit_training
from trl import SFTTrainer
# Configure 4-bit quantization
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4", # NormalFloat4 quantization
bnb_4bit_use_double_quant=True, # double quantization, more savings
bnb_4bit_compute_dtype=torch.bfloat16 # Compute in bfloat16 for speed
)
# Load model in 4-bit
model = AutoModelForCausalLM.from_pretrained(
model_name,
quantization_config=bnb_config,
device_map="auto"
)
# k-bit training prep: gradient checkpointing on 4-bit weights
model = prepare_model_for_kbit_training(model)
# Rest is the same as step 2: LoRA config, load data, train with SFTTrainer
lora_config = LoraConfig(r=8, lora_alpha=16, ...)
trainer = SFTTrainer(model=model, peft_config=lora_config, ...)
trainer.train()This configuration loads a base model in 4-bit NF4 with double quantization, uses bfloat16 for computation, and prepares the model for k-bit training before applying LoRA. section 5.1.6
Trading training time for GPU memory
For Qwen3-4B, QLoRA uses approximately 4–6 GB of GPU memory, compared with 8–12 GB for regular LoRA. On the same 24 GB GPU, a run over 450 examples for three epochs took 14–15 minutes with QLoRA versus 11 minutes with LoRA; quantization and dequantization add overhead. section 5.1.6
Table 5.5 Typical memory requirements for fine-tuning a model of various sizes
|
Model Size
|
Full
Fine-Tuning (VRAM)
|
LoRA (VRAM)
|
QLoRA (VRAM)
|
Notes
|
|
7 B
|
~60 to 67 GB
|
16 to 20 GB
|
8 to 10 GB
|
Fits on a
single 24 GB GPU using LoRA. QLoRA enables larger batch sizes or longer
contexts.
|
|
13 B
|
~120 to 130 GB
|
30 to 36 GB
|
10 to 18 GB
|
LoRA often
requires two 24 GB GPUs for 13B; QLoRA may fit on one high-end consumer GPU.
|
|
65 B
|
600 GB+
(multi-GPU)
|
~146 GB
|
~48 GB+
|
65B LoRA needs
a multi-GPU node (e.g., 8x H100). QLoRA fits 65B on a single 48 GB GPU, the
QLoRA paper’s headline result [2]; an 80 GB H100 or H200 gives headroom for
longer contexts.
|
Compare typical VRAM requirements across full fine-tuning, LoRA, and QLoRA for 7B, 13B, and 65B models. Requirements vary with the model and workload. section 5.5
Choose regular LoRA when it fits the available hardware; use QLoRA when memory constraints justify slower training. Training-time bitsandbytes quantization differs from serving optimization: a usual deployment path merges the adapter into the full-precision base, then quantizes the merged model. Formats such as GPTQ or AWQ typically offer higher inference throughput. section 5.1.6 section 5.5





















