The root mean square (RMS) of a vector is the square root of the mean of its squared values. RMSNorm uses the inverse RMS, with a small ε for stability, to scale each input element; it does not subtract the mean. section 4.1.4
RMSNorm scales each element by the inverse RMS and its learned per-element factor γ. section 4.1.4
On a GPU, RMSNorm follows a reduce-then-map pattern: first compute the row’s mean square and inverse RMS, then reread the input to normalize and apply γ. This two-pass design avoids keeping a large row in registers while its statistic is computed. section 4.1.4
