liveBook · Manning

Floating-point addition

As explained in GPU Programming with Triton

Floating-point addition is addition performed on finite-precision floating-point representations with rounding of the computed result.

Floating-point addition rounds finite-precision results, so regrouping a sum can change its value. GPU tree reductions regroup additions to combine values in parallel; their results can differ from a sequential sum in the last few bits, even when each result is numerically close. section 4.3 section 4.1.1

print((0.1 + 1e20) - 1e20)  # 0.0
print(0.1 + (1e20 - 1e20))  # 0.1

These two groupings of the same values produce different results because adding 0.1 to 1e20 rounds away the smaller value. section 4.1.1

Reduction order can vary with tile shape, warp count, or atomic update order. Higher-precision accumulation, such as summing fp16 or bf16 inputs into fp32, can help limit numerical error. section 4.3 section 4.1.1

How GPU Programming with Triton covers it

How deep it goes
Defines non-associativity, demonstrates rounding with values, and connects regrouping to GPU reduction order.
Examples
Demonstrates regrouping with 0.1 and 1e20, then discusses tree reductions and atomic sums.
What it assumes
Builds from the definition of associativity and finite-precision rounding before relating them to reduction trees.
What you can do after
Apply higher-precision accumulation, such as summing fp16 or bf16 inputs into fp32, to help limit reduction error.
How it connects
Links floating-point addition to parallel reduction, rounding error, atomic accumulation, and softmax.

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.