Floating-point addition rounds finite-precision results, so regrouping a sum can change its value. GPU tree reductions regroup additions to combine values in parallel; their results can differ from a sequential sum in the last few bits, even when each result is numerically close. section 4.3 section 4.1.1
print((0.1 + 1e20) - 1e20) # 0.0 print(0.1 + (1e20 - 1e20)) # 0.1
These two groupings of the same values produce different results because adding 0.1 to 1e20 rounds away the smaller value. section 4.1.1
Reduction order can vary with tile shape, warp count, or atomic update order. Higher-precision accumulation, such as summing fp16 or bf16 inputs into fp32, can help limit numerical error. section 4.3 section 4.1.1
