liveBook · Manning

Accumulation precision

As explained in GPU Programming with Triton

Higher-precision accumulation performs an aggregate computation in a numerical format with greater precision than the input format.

When reducing fp16 or bf16 values, accumulating in fp32 reduces rounding error and usually improves stability for large reductions. In Triton, set the accumulator type with the dtype argument to tl.sum; making this choice explicit is good practice. Higher precision helps with rounding but does not make floating-point addition associative, so regrouped reductions can still differ in their last bits. section 4.1.1 section 4.3

print((0.1 + 1e20) - 1e20)  # 0.0
print(0.1 + (1e20 - 1e20))  # 0.1

These two groupings produce different results, illustrating why reduction order can affect floating-point sums. section 4.1.1

How GPU Programming with Triton covers it

How deep it goes
Defines fp32 accumulation for lower-precision inputs and identifies Triton's tl.sum dtype argument, without promising bitwise reproducibility.
Examples
Illustrates order-dependent rounding with two groupings of 0.1 and 1e20, and cites fp16 or bf16 inputs accumulated in fp32.
What you can do after
Apply explicit fp32 accumulation to fp16 or bf16 reductions using Triton's tl.sum dtype argument.
How it connects
Connects reductions with floating-point associativity, rounding, and numerically sensitive operations such as softmax.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.