liveBook · Manning

Per-token normalization

As explained in Reinforcement Learning from Human Feedback

Per-token normalization averages losses across all active tokens so that every token contributes equally.

Per-token normalization aggregates policy losses over active completion tokens, giving every token equal gradient weight. Longer sequences therefore exert proportionally more influence than shorter ones. section 6.3.2

For per-token losses ℓᵢ,ₜ, batch size B, and completion lengths |aᵢ|, compute the sum of all active-token losses divided by the total number of active tokens. Prompt positions remain excluded by the completion mask. section 6.3.2

Unequal sequence lengths

A five-token sequence with mean loss 2.8 and a ten-token sequence with mean loss 1.9 produce a per-token loss of (14 + 19) / 15 = 2.2. Every active token receives equal gradient magnitude, unlike per-sequence normalization. section 6.3.2

Equation 6.47
\[\label{eq:loss_per_token} L = \frac{\sum_{i=1}^{B} \sum_{t=1}^{|a_i|} \ell_{i,t}}{\sum_{i=1}^{B} |a_i|}.\]

Formalizes normalization across all active tokens in the batch. section 6.3.2

How Reinforcement Learning from Human Feedback covers it

How deep it goes
Defines the batch objective, masking rule, and sequence-length trade-off, but does not derive the policy-gradient loss itself.
Examples
Compares five-token and ten-token loss sequences using explicit sums, means, and resulting gradient weights.
What it assumes
Assumes per-token losses, completion lengths, batches, and masks; clarifies that prompt tokens are excluded.
What you can do after
Apply it via masked tensor reduction to give each active completion token equal loss weight.
How it connects
Contrasts with per-sequence normalization and fixed-length normalization, and uses completion masks within policy-gradient training.
Cover of Reinforcement Learning from Human Feedback
liveBookmanning.com
Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.