When reducing fp16 or bf16 values, accumulating in fp32 reduces rounding error and usually improves stability for large reductions. In Triton, set the accumulator type with the dtype argument to tl.sum; making this choice explicit is good practice. Higher precision helps with rounding but does not make floating-point addition associative, so regrouped reductions can still differ in their last bits. section 4.1.1 section 4.3
print((0.1 + 1e20) - 1e20) # 0.0 print(0.1 + (1e20 - 1e20)) # 0.1
These two groupings produce different results, illustrating why reduction order can affect floating-point sums. section 4.1.1
