An exponent allocates bits to magnitude in a floating-point number. It controls the binary decimal point, determining dynamic range: more exponent bits allow values to span a wider interval, from very small to very large. Mantissa bits instead control precision, so each format balances range against representational detail. section 5.5.3
The allocation becomes concrete across common numerical formats:
- FP32 uses 8 exponent bits and 23 mantissa bits, providing broad dynamic range and high precision. section 5.5.3
- FP16 uses 5 exponent bits and 10 mantissa bits, reducing memory while narrowing range and precision and increasing overflow or underflow risk. section 5.5.3
- BF16 retains FP32's 8 exponent bits but reduces the mantissa to 7 bits, preserving dynamic range at lower precision. section 5.5.3
- FP8 E5M2 favors range with 5 exponent bits, while E4M3 favors precision with 4 exponent bits and 3 mantissa bits. section 5.6.4
Local scaling changes the FP8 choice
A conventional mixed-precision design uses E4M3 for forward computation and E5M2 for gradients, whose outliers may exceed E4M3's range. Fine-grained quantization scales small, independent blocks, limiting local outlier effects and enabling uniform E4M3 for forward and backward operations. The design effectively shares exponent capacity across grouped elements while retaining more mantissa bits. section 5.6.4
Figure 5.18 A comparison of FP32 and FP16.

Compare how FP32 and FP16 distribute bits between exponent and mantissa. section 5.5.3
Figure 5.19 A comparison of FP32 and BFloat16.

Compare BF16's FP32-sized exponent with its reduced mantissa. section 5.5.3





