liveBook · Manning

Reduce-then-map

As explained in GPU Programming with Triton

A computation pattern that first reduces values to a statistic and then uses that statistic to transform each element.

Reduce-then-map computes a statistic over a vector or row, then uses that shared statistic to transform each element. The reduction must finish before the outputs can be produced. section 4.1.4

For large rows, the dependency shapes the implementation:

  • A two-pass kernel first reduces the input, then reloads it to produce outputs. section 4.1.4
  • Keeping the row in registers or shared memory can avoid reloading, but is limited by available capacity; two passes preserve a simple design and occupancy. section 4.1.4

RMSNorm

Compute the mean of squared inputs and derive an inverse RMS scalar; then reload each input and multiply it by that scalar and its learned scale. section 4.1.4

Figure 4.6 The reduce-then-map pattern in RMSNorm. Pass 1 reads the input, squares each element, and reduces to compute the inverse RMS. Pass 2 re-reads the input and applies the normalization with the learned scale γ. Two passes are necessary because the statistic must be fully computed before any output can be written
CH04 F06 Fartale

Shows the two passes in RMSNorm: reduce to the inverse RMS, then reload and scale the input. section 4.1.4

How GPU Programming with Triton covers it

How deep it goes
Explains the two-pass dependency and storage trade-off, then demonstrates single-vector and batched Triton kernels.
Examples
Uses RMSNorm’s sum of squares, inverse RMS, and learned scale in single-vector and batched kernels.
What you can do after
Apply it via Triton to implement RMSNorm with masked loads, a row reduction, a second input read, and elementwise scaling.
How it connects
Connects RMSNorm and softmax through row statistics, and relates the same dependency constraint to attention.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.