A GPU communication hierarchy matches cooperation distance to mechanism: lanes within a warp exchange values through registers, warps within a block use shared memory and synchronization, and blocks coordinate through global memory. section 4.0 section 4.1
For a tree reduction, each level of cooperation has a corresponding mechanism:
- Lanes in one warp exchange partial values through register-level shuffle instructions. section 4.1
- Warps in one block communicate through shared memory, with synchronization to prevent reads racing ahead of writes. section 4.1
- Blocks cannot synchronize within a single kernel launch; coordinating across blocks uses global memory, usually with another launch or atomic operations. section 4.1
Triton reduction primitives combine values in a tensor local to one program instance; a grid-wide reduction needs explicit orchestration, such as a second pass over partial results or atomic updates. section 4.3
Figure 4.2 The communication hierarchy for reductions on a GPU. Lanes within a warp exchange values through fast register shuffles. Warps within a block communicate through shared memory with explicit barriers. Blocks cannot synchronize inside a single kernel and must coordinate through global memory
Trace how a reduction maps from warp-local shuffles to shared-memory barriers and then to global-memory coordination across blocks. section 4.1
