QLoRA (Quantized LoRA) is a parameter-efficient fine-tuning technique that keeps a pretrained base model frozen, stores it in 4-bit form, and trains 16-bit LoRA adapter matrices. It reduces the memory burden of specialization while retaining low-rank adaptation, making large-model fine-tuning more practical on constrained hardware. section 3.5.2 section 1.4.2
The central trade-off is compression versus effective adaptation. Quantization alone can reduce performance, so QLoRA combines compressed frozen weights with trainable low-rank updates. Optimizer states remain 32-bit for trainable parameters; when GPU memory is tight, CPU paging can offload states and components temporarily. section 3.5.2
A practical QLoRA workflow is:
- Load a pretrained classification model with a 4-bit quantization configuration; the example uses double quantization, NF4, bfloat16 computation, and automatic device placement. section 3.5.2
- Prepare the model for k-bit training and enable gradient checkpointing to further manage memory. section 3.5.2
- Tokenize and split labeled data, then attach a LoRA configuration targeting selected modules while retaining the task classifier. section 3.5.2
- Train with evaluation and checkpointing settings, using a paged 8-bit AdamW optimizer in the illustrated configuration. section 3.5.2
Sentiment classification with a quantized base model
A Rotten Tomatoes dataset is mapped to NEGATIVE and POSITIVE labels, tokenized to a maximum length of 512, and used to fine-tune bert-base-uncased. The base model is loaded in 4-bit form; LoRA targets query and value modules with rank 16, while the classifier is retained for sequence classification. section 3.5.2
Figure 3.12 compares LoRA and QLoRA side by side. QLoRA uses a quantized model instead of the original model. It can also offload optimizer states to CPU memory when GPU memory is tight.

Compare the storage layout directly: LoRA retains a 16-bit frozen base model, whereas QLoRA uses a 4-bit base model while keeping 16-bit adapters and 32-bit optimizer states. section 3.5.2






