liveBook · Manning

Inverse RMS

As explained in GPU Programming with Triton

The inverse RMS is the reciprocal of a vector's root mean square and serves as its normalization factor.

The inverse RMS is the reciprocal square root of a vector’s mean squared value plus a small ε. RMSNorm multiplies each input element by this scalar, then applies a learned per-element scale; unlike LayerNorm, it does not subtract the mean. section 4.1.4

\[y_i = \frac{x_i}{\sqrt{\frac{1}{n} \sum_{j=1}^{n} x_j^2 + \epsilon}} \cdot \gamma_i\]

The equation shows the inverse RMS scaling each input element before the learned scale is applied. section 4.1.4

Computing the scalar requires reducing across the full vector before producing normalized outputs. Triton kernels therefore use a reduce-then-map structure: accumulate squared values, calculate the inverse RMS, then re-read and transform the input. section 4.1.4

How GPU Programming with Triton covers it

How deep it goes
Defines the reciprocal-root statistic and explains its two-pass reduction and mapping implementation in Triton.
Examples
Uses single-vector and batched RMSNorm kernels to demonstrate inverse RMS computation.
What you can do after
Apply it via a Triton RMSNorm kernel that reduces squared inputs and normalizes with the resulting scalar.
How it connects
Connects RMSNorm to LayerNorm, reduce-then-map, softmax, and attention.

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.