The inverse RMS is the reciprocal square root of a vector’s mean squared value plus a small ε. RMSNorm multiplies each input element by this scalar, then applies a learned per-element scale; unlike LayerNorm, it does not subtract the mean. section 4.1.4
The equation shows the inverse RMS scaling each input element before the learned scale is applied. section 4.1.4
Computing the scalar requires reducing across the full vector before producing normalized outputs. Triton kernels therefore use a reduce-then-map structure: accumulate squared values, calculate the inverse RMS, then re-read and transform the input. section 4.1.4
