liveBook · Manning

Rmsnorm

As explained in GPU Programming with Triton

RMSNorm normalizes activations using their root-mean-square magnitude without subtracting the mean.

RMSNorm scales each input element by the inverse root-mean-square of its vector or row, then applies a learned per-element scale. Unlike LayerNorm, it does not subtract the mean. A small ε stabilizes the reciprocal square root and prevents division by zero. section 4.1.4

\[y_i = \frac{x_i}{\sqrt{\frac{1}{n} \sum_{j=1}^{n} x_j^2 + \epsilon}} \cdot \gamma_i\]

The formula shows the sum-of-squares statistic shared across the vector and the learned scale applied to each output. section 4.1.4

A Triton implementation follows a reduce-then-map pattern: first accumulate squared inputs and compute the inverse RMS; then reread the inputs and apply the normalization and scale. The statistic must be known before outputs can be written. For large rows, rereading trades extra global-memory bandwidth for lower register and shared-memory use, helping preserve occupancy. section 4.1.4

Vector and batched kernels

A single-vector kernel loops over input blocks in two passes. A batched 2D kernel assigns a program instance to each row and performs the same reduction and normalization for that row. section 4.1.4

RMSNorm shares its reduce-then-map structure with softmax: both compute a row statistic and broadcast it across the row. Softmax additionally needs a stability adjustment before exponentiation. section 4.3

How GPU Programming with Triton covers it

How deep it goes
Explains the formula, two-pass Triton implementation, and memory trade-off without detailing performance measurements.
Examples
Shows single-vector and batched 2D Triton kernels that reduce squared values before normalizing and scaling outputs.
What you can do after
Apply it via Triton kernels that reduce each vector or row’s squared values, then normalize and scale its outputs.
How it connects
Connects RMSNorm to LayerNorm through their normalization contrast and to softmax through their reduce-then-map structure.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.