liveBook · Manning

Two-pass computation

As explained in GPU Programming with Triton

Two-pass computation performs one pass to calculate a statistic and a separate pass to use that statistic for the final transformation.

Two-pass computation separates a reduction from the transformation that uses its result. In RMSNorm, the first pass reduces squared inputs to an inverse RMS value; the second rereads the inputs and applies that value and a learned scale. section 4.1.4

The two-pass structure trades extra input reads for simpler resource use:

  • Keeping a large row live while waiting for its statistic can exceed register capacity; caching it in shared memory is fast but capacity-limited. section 4.1.4
  • Rereading inputs costs bandwidth, but can preserve occupancy and keep the kernel simple for large rows. section 4.1.4
Figure 4.6 The reduce-then-map pattern in RMSNorm. Pass 1 reads the input, squares each element, and reduces to compute the inverse RMS. Pass 2 re-reads the input and applies the normalization with the learned scale γ. Two passes are necessary because the statistic must be fully computed before any output can be written
CH04 F06 Fartale

The diagram shows the reduction pass followed by the reread-and-transform pass. section 4.1.4

How GPU Programming with Triton covers it

How deep it goes
Traces RMSNorm from sum-of-squares reduction through inverse RMS calculation to reloading and scaling elements.
Examples
Uses single-vector and batched two-dimensional RMSNorm kernels to illustrate the pattern.
What you can do after
Build it from scratch as a Triton RMSNorm kernel with separate reduction and output passes.
How it connects
Connects RMSNorm with reductions, softmax, and attention through statistics that must be computed before outputs.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.