chapter four
4 Reduction Patterns
This chapter covers
- How reduction trees map onto the GPU communication hierarchy
- Numerical issues in floating-point reductions and how to mitigate them
- Triton’s reduction primitives and the limits of their synchronization scope
- Reduce-then-map kernels like RMSNorm and numerically stable softmax
We now turn to one of the most fundamental operations in parallel computing: reduction. Reductions appear throughout machine learning: summing gradients during backpropagation, computing the denominator in softmax, or accumulating statistics for normalization layers. Operations like normalization and probability distributions rely on global statistics of this kind. The denominator of a softmax calculation is a single scalar value, but it is derived from every element in a tensor row. When that row is distributed across thousands of executing threads, no individual thread has enough information to compute its final output using only the data in its local registers. The arithmetic itself forces the threads to cooperate.