The mantissa, also called the significand, stores the significant digits of a floating-point number. It determines precision: more mantissa bits produce more significant digits and smaller gaps between representable values. The exponent instead controls dynamic range, while the sign bit records positivity or negativity. section 5.5.3
Compare the precision and range trade-offs across common numerical formats:
- FP32 allocates 23 mantissa bits, alongside 8 exponent bits, providing high precision and broad dynamic range. section 5.5.3
- FP16 allocates 10 mantissa bits and 5 exponent bits, reducing memory while narrowing both precision and range; overflow and underflow become more likely. section 5.5.3
- BF16 keeps FP32's 8 exponent bits but reduces the mantissa to 7 bits, preserving dynamic range at lower precision. section 5.5.3
- FP8 E5M2 favors range with 5 exponent bits and 2 mantissa bits, whereas E4M3 favors precision with 4 exponent bits and 3 mantissa bits. section 5.6.4
Mantissa bits become especially important in quantized training. A conventional FP8 strategy uses E4M3 for the forward pass and E5M2 for gradients, where large outliers can overflow the narrower range. Fine-grained quantization scales small, independent blocks, limiting outlier effects and allowing E4M3's higher precision to be used uniformly for forward and backward operations. section 5.6.4
Figure 5.18 A comparison of FP32 and FP16.

Use the FP32-versus-FP16 comparison to visualize how reducing mantissa and exponent bits changes precision and dynamic range. section 5.5.3
Figure 5.19 A comparison of FP32 and BFloat16.

Use the FP32-versus-BF16 comparison to see how BF16 preserves exponent capacity while sacrificing mantissa precision. section 5.5.3






