Numerical overflow occurs when a computed value exceeds the finite range of its number format. In float32, exp(88.7) is already about 3.3 × 10³⁸; in float16, exp(12) overflows. Infinities can propagate into NaNs and destabilize training or inference. section 4.2.1
Softmax avoids this problem by subtracting the vector maximum before exponentiation. Because adding the same constant to every input leaves softmax unchanged, this shift preserves the result while making the largest exponent exp(0) = 1. The maximum is found by a reduction, then broadcast across the vector. section 4.2.1 section 4.1.1
Figure 4.7 (Left) Dynamic range and precision comparison between FP32 and FP16. FP32 covers a far wider range with finer spacing between representable values. (Right) Bit-level breakdown showing how both formats encode the same number, with FP16 producing a less precise result due to its shorter mantissa.
Compare the dynamic range and precision of FP32 and FP16, which affect how readily computations overflow. section 4.2.1







