Floating-point precision describes how finely a format distinguishes nearby values. Finite precision causes rounding: a small value can disappear when added to a much larger one. FP32 offers finer spacing and a wider range than FP16; FP16’s shorter mantissa rounds more aggressively. section 4.1.1 section 4.2.1
Because floating-point addition is not truly associative, regrouping a reduction can change its result. Parallel reduction trees, tile shapes, and warp counts can therefore produce numerically close but not bitwise-identical results. Small differences can matter when a reduction feeds a sensitive operation such as softmax. section 4.1.1
Regrouping changes the result
With the same values, (0.1 + 1e20) − 1e20 evaluates to 0.0, while 0.1 + (1e20 − 1e20) evaluates to 0.1. The first grouping rounds away 0.1. section 4.1.1
Figure 4.7 (Left) Dynamic range and precision comparison between FP32 and FP16. FP32 covers a far wider range with finer spacing between representable values. (Right) Bit-level breakdown showing how both formats encode the same number, with FP16 producing a less precise result due to its shorter mantissa.
Compare FP32 and FP16 spacing, dynamic range, and bit-level representation. section 4.2.1












