liveBook · Manning

Root mean square

As explained in GPU Programming with Triton

The root mean square is the square root of the arithmetic mean of squared values.

The root mean square (RMS) of a vector is the square root of the mean of its squared values. RMSNorm uses the inverse RMS, with a small ε for stability, to scale each input element; it does not subtract the mean. section 4.1.4

\[y_i = \frac{x_i}{\sqrt{\frac{1}{n} \sum_{j=1}^{n} x_j^2 + \epsilon}} \cdot \gamma_i\]

RMSNorm scales each element by the inverse RMS and its learned per-element factor γ. section 4.1.4

On a GPU, RMSNorm follows a reduce-then-map pattern: first compute the row’s mean square and inverse RMS, then reread the input to normalize and apply γ. This two-pass design avoids keeping a large row in registers while its statistic is computed. section 4.1.4

How GPU Programming with Triton covers it

How deep it goes
Explains the normalization equation and two-pass GPU reduction, then demonstrates single-vector and batched Triton kernels.
Examples
Demonstrates RMSNorm on a single vector and on rows of a batched two-dimensional input.
What you can do after
Apply it via Triton by reducing squared values, computing inverse RMS, and scaling the input in a second pass.
How it connects
Contrasts RMSNorm with LayerNorm and relates its reduce-then-map structure to softmax and attention.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.