Reduce-then-map computes a statistic over a vector or row, then uses that shared statistic to transform each element. The reduction must finish before the outputs can be produced. section 4.1.4
For large rows, the dependency shapes the implementation:
- A two-pass kernel first reduces the input, then reloads it to produce outputs. section 4.1.4
- Keeping the row in registers or shared memory can avoid reloading, but is limited by available capacity; two passes preserve a simple design and occupancy. section 4.1.4
RMSNorm
Compute the mean of squared inputs and derive an inverse RMS scalar; then reload each input and multiply it by that scalar and its learned scale. section 4.1.4
Figure 4.6 The reduce-then-map pattern in RMSNorm. Pass 1 reads the input, squares each element, and reduces to compute the inverse RMS. Pass 2 re-reads the input and applies the normalization with the learned scale γ. Two passes are necessary because the statistic must be fully computed before any output can be written
Shows the two passes in RMSNorm: reduce to the inverse RMS, then reload and scale the input. section 4.1.4
