liveBook · Manning

Communication hierarchy

As explained in GPU Programming with Triton

A communication hierarchy is an ordered set of communication mechanisms whose scope and storage locations vary across threads, warps, blocks, and the grid.

A GPU communication hierarchy matches cooperation distance to mechanism: lanes within a warp exchange values through registers, warps within a block use shared memory and synchronization, and blocks coordinate through global memory. section 4.0 section 4.1

For a tree reduction, each level of cooperation has a corresponding mechanism:

  • Lanes in one warp exchange partial values through register-level shuffle instructions. section 4.1
  • Warps in one block communicate through shared memory, with synchronization to prevent reads racing ahead of writes. section 4.1
  • Blocks cannot synchronize within a single kernel launch; coordinating across blocks uses global memory, usually with another launch or atomic operations. section 4.1

Triton reduction primitives combine values in a tensor local to one program instance; a grid-wide reduction needs explicit orchestration, such as a second pass over partial results or atomic updates. section 4.3

Figure 4.2 The communication hierarchy for reductions on a GPU. Lanes within a warp exchange values through fast register shuffles. Warps within a block communicate through shared memory with explicit barriers. Blocks cannot synchronize inside a single kernel and must coordinate through global memory
CH04 F02 Fartale

Trace how a reduction maps from warp-local shuffles to shared-memory barriers and then to global-memory coordination across blocks. section 4.1

How GPU Programming with Triton covers it

How deep it goes
Maps warp, block, and grid communication to mechanisms, and identifies the lack of in-kernel synchronization across blocks.
Examples
Uses an eight-value tree reduction and reductions for softmax denominators and normalization statistics.
What you can do after
Evaluate which communication mechanism and orchestration a reduction requires at each GPU execution scope.
How it connects
Connects reductions with GPU warps, shared memory, global memory, synchronization, and Triton program instances.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.