Precision is the fineness with which a numerical format preserves significant digits and separates nearby values. In floating-point formats, mantissa bits determine precision: more mantissa bits produce more significant digits and smaller gaps between representable numbers. Exponent bits instead determine dynamic range, or how large and small a value the format can represent. section 5.5.3
Reducing bit-width reduces the palette of representable numbers. FP32 uses 23 mantissa bits, whereas FP16 uses 10, so FP16 saves memory but provides lower precision and a smaller range. BF16 keeps FP32's 8 exponent bits and therefore its broad range, but uses only 7 mantissa bits and has lower precision than FP16. section 5.5.1 section 5.5.3
Precision participates in practical format and training trade-offs:
- FP16 can overflow or underflow because reducing exponent bits limits its dynamic range as well as its precision. section 5.5.3
- FP8 E5M2 favors range with 5 exponent bits and 2 mantissa bits, while E4M3 favors precision with 4 exponent bits and 3 mantissa bits. section 5.6.4
- Fine-grained quantization scales small independent blocks, reducing outlier effects and preventing values from being excessively squashed into a narrow scale. section 5.6.4
Uniform E4M3 in training
A conventional mixed-precision design uses E4M3 for the forward pass and E5M2 for backward gradients, where outliers may overflow E4M3. With fine-grained scaling mitigating local outliers, a training design can use the higher-precision E4M3 format uniformly for forward and backward operations. section 5.6.4
Figure 5.15 A visual comparison of a number represented in 32-bit floating-point (FP32) versus 16-bit floating-point (FP16). The diagram illustrates the reduction in the number of bits allocated for the exponent and the mantissa, which results in lower precision and a smaller memory footprint.

Compare FP32 and FP16 visually by inspecting how fewer exponent and mantissa bits reduce precision and memory footprint. section 5.5.1












