Low-rank adaptation trains small, full-precision adapters while keeping pretrained weights frozen. In QLoRA, those base weights are quantized to 4-bit NF4, reducing the memory needed to fine-tune a model. section 8.4
Listing 8.1 Loading Llama-2-7B in NF4 with QLoRA adapters
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4", #A
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True, #B
)
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-2-7b-hf",
quantization_config=bnb_config,
device_map="auto",
)
lora_config = LoraConfig(
r=16, lora_alpha=32,
target_modules=["q_proj", "k_proj", "v_proj", #C
"o_proj", "gate_proj",
"up_proj", "down_proj"],
lora_dropout=0.05, bias="none",
task_type="CAUSAL_LM",
)
model = get_peft_model(
prepare_model_for_kbit_training(model), lora_config
)
Configuration example for loading a Llama-2-7B base in NF4 and attaching rank-16 LoRA adapters. section 8.4
Training memory
A measured Llama-2-7B QLoRA run uses 12.31 GB peak GPU memory, including optimizer states, gradients, and activations.
















