Nonuniform quantization maps values to discrete levels separated by variable-sized intervals. Adapting interval widths to the data distribution can reduce quantization error, especially where precision strongly affects model performance. section 8.3.3
Use nonuniform quantization to weigh accuracy against implementation and training costs:
- Compared with uniform quantization, it can tailor intervals differently across weights or model layers, potentially improving performance while requiring a more distribution-aware design. section 8.3.3
- Quantization-aware training retrains with quantized weights and activations, while posttraining quantization avoids retraining but generally supports less aggressive precision reduction. section 8.3.3
Quantized model loading
A bitsandbytes configuration loads Falcon 7B with 4-bit fp4 weights and bfloat16 computation, illustrating a practical low-precision deployment setup.
Table 8.1 Comparison of data types
|
Data type
|
Bit width
|
Range
|
Memory reduction from FP32
|
|---|---|---|---|
|
float32
|
32 bits
|
≈ ± 3.4 × 10^{38}
|
–
|
|
float16
|
16 bits
|
≈ ± 6.5 × 10^{4}
|
50%
|
|
int8
|
8 bits
|
-128 to 127
|
75%
|
|
int4
|
4 bits
|
-8 to 7
|
87.5%
|
Compare bit widths, numeric ranges, and memory reductions to place interval design within broader precision and deployment trade-offs. section 8.3.3
