Quantization converts model weights and activations from higher-precision formats such as float32 into lower-precision representations such as float16, int8, or int4. Lower bit depth reduces memory use and can improve computational efficiency, but may reduce accuracy or fail to represent values outside the smaller format’s range. section 8.3.3
The main deployment trade-offs are:
- float16 uses 16 bits and reduces memory by 50% relative to float32. section 8.3.3
- int8 uses 8 bits and reduces memory by 75%, while int4 uses 4 bits and reduces it by 87.5%. section 8.3.3
- Lower precision supports more efficient inference, embedded deployment, and fine-tuning a 7B model within limited hardware resources. section 8.3.3
- Reduced precision can lose accuracy and cannot represent very small or very large values outside the target format’s range. section 8.3.3
Loading Falcon 7B in 4-bit precision
Using bitsandbytes with PyTorch, configure a 4-bit model load by setting load_in_4bit=True, disabling 8-bit loading, selecting the fp4 quantization type, and using bfloat16 for computation. Load the tokenizer and model through the quantization configuration, then compare initial, final, and peak GPU memory. The illustrated quantized load uses about 4.07 GB, compared with about 25.88 GB without quantization. section 8.3.3
Table 8.1 Comparison of data types
|
Data type
|
Bit width
|
Range
|
Memory reduction from FP32
|
|---|---|---|---|
|
float32
|
32 bits
|
≈ ± 3.4 × 10^{38}
|
–
|
|
float16
|
16 bits
|
≈ ± 6.5 × 10^{4}
|
50%
|
|
int8
|
8 bits
|
-128 to 127
|
75%
|
|
int4
|
4 bits
|
-8 to 7
|
87.5%
|
Compare bit widths, numeric ranges, and memory reductions for float32, float16, int8, and int4. section 8.3.3
