Quantization represents higher-precision numerical values with lower-bit values, trading limited per-value precision for substantially lower memory use and computational cost. For large language models, this trade makes large-scale training and inference more affordable while often preserving overall model performance. section 5.5 section 5.5.1
A basic symmetric quantization pass follows these steps:
- Find the maximum absolute value α in the input vector or tensor. section 5.5.4
- Map the range −α to +α onto a target range such as −127 to 127, using scale = 127 / α. section 5.5.4
- Multiply each value by the scale and round it; for example, −7.59 becomes −89 when α = 10.8. section 5.5.4
- Dequantize by dividing the integer representation by the same scale, recovering an approximation of the original value. section 5.5.4
Modern low-precision training uses more than one precision. Core computations can use FP8, while master weights and optimizer states remain in higher precision for numerical stability. Fine-grained quantization assigns separate scales to small activation and weight blocks, reducing the effect of outliers. Intermediate results can accumulate in FP32, and online scaling can derive factors from the current batch rather than historical data. section 5.7
Memory trade-off for a 70-billion-parameter model
Quantizing weights from 32-bit to 16-bit reduces the stated storage requirement from 280 GB to 140 GB. Moving to lower precision reduces data movement and can allow the model to run on hardware with less memory, at the cost of some numerical precision. section 5.5.2
Figure 5.21 The scaling process for quantization. The original range of the FP32 tensor is mapped to the target range of the INT8 format.

Visualizes how an FP32 tensor range maps onto the INT8 target range during scaling. section 5.5.4





















