liveBook · Manning

Per-sequence normalization

As explained in Reinforcement Learning from Human Feedback

Per-sequence normalization averages each sequence's token losses so that every sequence contributes equally regardless of length.

Per-sequence normalization averages each completion’s token losses before averaging across the batch, giving every sequence equal influence regardless of length. For losses ℓᵢ,ₜ and completion length |aᵢ|, L = (1/B) Σᵢ (1/|aᵢ|) Σₜ ℓᵢ,ₜ. section 6.3.2

Its optimization behavior differs from alternative loss reductions:

  • Shorter sequences receive larger per-token gradients than longer sequences when sequence-level losses are equally weighted. section 6.3.2
  • Per-token normalization weights every token equally, while fixed-length normalization preserves sequence-level averaging but uses a constant Lmax denominator. section 6.3.2
  • The method emphasizes outcomes over individual tokens but can over-weight short strategies or downweight strategies that naturally require more tokens. section 6.3.2

Unequal token influence

For losses [1,1,1,1,10] and [1,1,1,1,1,1,1,1,1,10], the sequence means are 2.8 and 1.9; per-sequence normalization averages them to 2.35, giving short-sequence tokens stronger gradients. section 6.3.2

Equation 6.46
\[\label{eq:loss_per_sequence} L = \frac{1}{B} \sum_{i=1}^{B} \frac{1}{|a_i|} \sum_{t=1}^{|a_i|} \ell_{i,t}.\]

Equation 6.46 gives the formal reduction; the masked implementation first averages active completion-token losses per sequence, then averages sequences. section 6.3.2

How Reinforcement Learning from Human Feedback covers it

How deep it goes
Defines the reduction, provides masked code and examples, and stops short of deriving the underlying policy-gradient objective.
Examples
Uses two five- and ten-token loss sequences, their means, and masked reduction code to expose unequal gradient weighting.
What it assumes
Assumes per-token losses, completion lengths, batch dimensions, and masks that exclude prompt tokens from learning.
What you can do after
Apply it via masked sequence-wise averaging in GRPO or some PPO implementations.
How it connects
Connects policy gradients with per-token normalization, fixed-length normalization, completion masks, and sequence-length gradient bias.
Cover of Reinforcement Learning from Human Feedback
liveBookmanning.com
Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.