QLoRA (quantized low-rank adaptation) is a parameter-efficient way to fine-tune a large language model when GPU memory makes full fine-tuning impractical. It combines LoRA’s small trainable adapters with 4-bit storage for the base model’s weights. The base weights stay frozen; training updates the adapters rather than every model parameter. section 5.3.3
The two savings address different costs. LoRA trains roughly 0.1–1% of parameters. Four-bit quantization reduces the memory needed to hold model weights compared with 16- or 32-bit storage; the configuration below uses NF4. The trade-off is not a promise of identical results: the comparison places QLoRA at roughly 90–95% of full-fine-tuning performance, with the lowest memory use of the three approaches compared. section 5.3.3
A customer-support assistant gives the workflow a concrete shape:
- Prepare focused instruction–response pairs, such as customer questions about orders, shipping, returns, and refunds. A demonstration uses 1,000 examples from the Bitext support dataset. section 5.3.4
- Load the base model with 4-bit NF4 quantization, prepare it for low-bit training, and attach LoRA adapters to the query and value projection modules. section 5.3.4
- Format each pair as a conversational prompt, train with SFTTrainer, then try realistic support questions. For evaluation, configure an evaluation dataset and track evaluation loss rather than relying on a few plausible replies. section 5.3.4
base_model = "NousResearch/Llama-2-7b-chat-hf" #1
bnb_config = BitsAndBytesConfig( #1
load_in_4bit=True, #1
bnb_4bit_quant_type="nf4", #1
bnb_4bit_compute_dtype=torch.float16, #1
)
model = AutoModelForCausalLM.from_pretrained(
base_model,
quantization_config=bnb_config,
device_map="auto",
trust_remote_code=True, # Required for some models that include custom
code in their repository. Only enable for models from trusted sources.
)
model = prepare_model_for_kbit_training(model)
tokenizer = AutoTokenizer.from_pretrained(base_model)
tokenizer.pad_token = tokenizer.eos_token
This loading configuration shows the memory-saving half of QLoRA: 4-bit NF4 weights with float16 computation, followed by preparation for low-bit training. Start from an unquantized base-model source; loading it quantized for training is not the same as fine-tuning a model that was pre-quantized. section 5.3.4
peft_config = LoraConfig( #1
r=8, #2
lora_alpha=16, #3
target_modules=["q_proj", "v_proj"], #4
lora_dropout=0.1, #5
bias="none", #6
task_type="CAUSAL_LM" #7
)
model = get_peft_model(model, peft_config)
This adapter configuration shows the trainable half: LoRA targets q_proj and v_proj while the quantized base model remains frozen. section 5.3.4
Choose QLoRA when limited GPU resources rule out more memory-intensive fine-tuning but adapting a large model remains worthwhile. A 7B conversational model is used for a 24GB-GPU example; larger variants may need multiple GPUs or cloud infrastructure. After training, save the adapted model and tokenizer, then reload the adapter with the base model. Test it on the support requests it must handle, including questions outside a small demonstration sample. section 5.3.3 section 5.3.4
