Quantization represents model values with fewer bits to reduce memory use and, in some cases, computation. A practical fine-tuning setup can load a base model at 4-bit precision rather than the standard 32-bit representation, allowing a capable model to fit consumer hardware while retaining useful performance. section 2.1.2
In parameter-efficient fine-tuning (PEFT), quantization complements targeted updates rather than replacing them. It reduces the footprint of the frozen model while lightweight trainable components adapt it to a specialized task, making large-model adaptation more accessible and deployable under resource constraints. section 3.5 section 1.4.2
Use QLoRA to separate what is compressed from what remains precise:
- Store the frozen base-model weights at 4-bit precision, reducing their memory requirement fourfold relative to 16-bit storage. section 3.5.2
- Keep LoRA adapter matrices at 16-bit precision for training stability, while optimizer states remain 32-bit for trainable parameters. section 3.5.2
- When GPU memory is constrained, offload optimizer states and related components to CPU memory, then page them back to the GPU. section 3.5.2
4-bit QLoRA sentiment classifier
Configure 4-bit loading, double quantization, NF4 quantization type, and bfloat16 compute; load bert-base-uncased with that configuration; then prepare it for k-bit training, enable gradient checkpointing, and attach a LoRA configuration for Rotten Tomatoes classification. section 3.5.2
Quantization is a resource-performance trade-off, not a guarantee of unchanged quality: compression alone can degrade performance. QLoRA addresses that risk by pairing a quantized frozen base model with low-rank updates, aiming to preserve effective fine-tuning while lowering memory and compute demands. section 3.5.2 section 3.8
Figure 3.12 compares LoRA and QLoRA side by side. QLoRA uses a quantized model instead of the original model. It can also offload optimizer states to CPU memory when GPU memory is tight.

Compare LoRA and QLoRA storage choices: the adapter and optimizer precision stay the same, while QLoRA compresses the frozen base model and can use CPU paging. section 3.5.2





















