Quantization reduces the precision used to store model weights, lowering memory requirements. In QLoRA, weights use 4-bit precision instead of 16- or 32-bit precision; training calculations can still use higher precision, such as float16. section 5.3.3
QLoRA combines quantization with additional memory-saving techniques:
- NF4 format aims to preserve important weight information, while double quantization compresses the quantization settings. section 5.3.3
- The stated trade-off is substantial memory savings with some loss of detail; the example estimates about 75% less memory from 4-bit weight storage. section 5.3.3
These savings can make fine-tuning a 7-billion-parameter model possible on a 24GB consumer GPU, or a 70-billion-parameter model on one 80GB GPU. Quantization is presented as part of an approach to deploying models at production scale and cost. section 5.3.3 section 1.4.1
Table 5.2 Comparing different fine-tuning methods
|
Method
|
Parameter efficiency
|
Performance
|
Memory usage
|
When to choose
|
|---|---|---|---|---|
|
Full fine-tuning
|
Lowest (100%)
|
Highest
|
Highest
|
When you have multiple high-end GPUs and need maximum adaptation
|
|
LoRA
|
High (~0.1–1%)
|
Very high (95–99% of full)
|
Low
|
When you have a decent GPU and need strong performance
|
|
QLoRA
|
High (~0.1–1%)
|
High (90–95% of full)
|
Lowest
|
When you have limited GPU resources but need to fine-tune large models
|
Compare QLoRA’s memory use and performance with LoRA and full fine-tuning. section 5.3.3
