An operation is associative when regrouping its operands leaves the result unchanged. For real numbers, addition satisfies (a + b) + c = a + (b + c), which makes associative operations straightforward to divide into partial reductions. section 4.1.1
Floating-point addition is not truly associative: finite precision can make a regrouping change the result. Parallel tree reductions necessarily regroup additions, so results may differ slightly with the reduction order, tile shape, or number of participating warps. section 4.1.1 section 4.3
Regrouping changes a floating-point sum
These expressions use the same values but produce different results: (0.1 + 1e20) - 1e20 gives 0.0, while 0.1 + (1e20 - 1e20) gives 0.1. The small value is rounded away when added to the large partial sum. section 4.1.1
Regrouping matters especially when:
- Bitwise reproducibility is required across runs or configurations. section 4.1.1
- Atomic updates combine partial sums in hardware-dependent order. section 4.1.1
- A reduction feeds a numerically sensitive computation such as softmax. section 4.1.1






