RMSNorm scales each input element by the inverse root-mean-square of its vector or row, then applies a learned per-element scale. Unlike LayerNorm, it does not subtract the mean. A small ε stabilizes the reciprocal square root and prevents division by zero. section 4.1.4
The formula shows the sum-of-squares statistic shared across the vector and the learned scale applied to each output. section 4.1.4
A Triton implementation follows a reduce-then-map pattern: first accumulate squared inputs and compute the inverse RMS; then reread the inputs and apply the normalization and scale. The statistic must be known before outputs can be written. For large rows, rereading trades extra global-memory bandwidth for lower register and shared-memory use, helping preserve occupancy. section 4.1.4
Vector and batched kernels
A single-vector kernel loops over input blocks in two passes. A batched 2D kernel assigns a program instance to each row and performs the same reduction and normalization for that row. section 4.1.4
RMSNorm shares its reduce-then-map structure with softmax: both compute a row statistic and broadcast it across the row. Softmax additionally needs a stability adjustment before exponentiation. section 4.3
