liveBook · Manning

Learned scale vector

As explained in GPU Programming with Triton

A learned scale vector is a trainable vector of per-element multiplicative factors applied to normalized activations.

In RMSNorm, the learned scale vector γ supplies one trainable multiplicative factor per input element. After scaling the input by its shared inverse RMS, the kernel multiplies each normalized element by its corresponding γᵢ. RMSNorm does not subtract the mean. section 4.1.4

\[y_i = \frac{x_i}{\sqrt{\frac{1}{n} \sum_{j=1}^{n} x_j^2 + \epsilon}} \cdot \gamma_i\]

The equation shows γᵢ applied elementwise after the shared RMS normalization. section 4.1.4

How GPU Programming with Triton covers it

How deep it goes
Defines the scale’s role and shows it in two-pass single-vector and batched Triton kernels.
Examples
Demonstrates elementwise scaling in single-vector and batched 2D RMSNorm kernels.
What you can do after
Apply a learned elementwise scale in Triton RMSNorm kernels after computing inverse RMS.
How it connects
Connects the scale vector to RMSNorm and contrasts RMSNorm with LayerNorm.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.