liveBook · Manning

Loss aggregation

As explained in Reinforcement Learning from Human Feedback

Loss aggregation is the method of combining per-token or per-sequence losses into a scalar optimization loss.

Loss aggregation combines per-token or per-sequence losses into the scalar used for optimization. The normalization choice changes how sequence length influences gradients: averaging each sequence separately gives every completion equal weight, while averaging all valid tokens gives longer completions more influence. section 6.3.2

Common aggregation strategies include:

  • Per-sequence mean: average each completion’s masked token losses, then average across the batch. This gives each sequence equal weight. section 6.3.2
  • Per-token mean: sum all masked token losses and divide by the total number of valid tokens. Longer completions contribute more tokens and therefore more total influence. section 6.3.2
  • Fixed-length normalization: average each sequence’s loss using a shared maximum length, preserving equal sequence weighting while changing gradient scale for shorter completions. section 6.3.2

Why normalization changes weighting

Suppose one completion has five token losses [1, 1, 1, 1, 10], with mean 2.8, and another has ten losses [1, 1, 1, 1, 1, 1, 1, 1, 1, 10], with mean 1.9. Per-sequence averaging treats their means equally; per-token averaging weights the longer completion more heavily. With masked_mean, short sequences can receive larger per-token gradients than long sequences, although gradient accumulation can reverse the balance. section 6.3.2

Aggregation also reflects the RL formulation. MDP-style training assigns token-level advantages, rewards, and often KL penalties. Bandit-style training assigns one sequence-level advantage and broadcasts it across the completion. GRPO commonly combines a bandit-style advantage with a separate per-token KL term, while PPO and RLOO may incorporate KL into rewards before advantage computation. section 6.3.2 section 6.6

Equation 6.46
\[\label{eq:loss_per_sequence} L = \frac{1}{B} \sum_{i=1}^{B} \frac{1}{|a_i|} \sum_{t=1}^{|a_i|} \ell_{i,t}.\]

Use the per-sequence objective to inspect how masked token losses are normalized within each completion before batch averaging. section 6.3.2

How Reinforcement Learning from Human Feedback covers it

How deep it goes
Derives three normalization objectives, provides masked implementations, and stops short of prescribing one universally optimal strategy.
Examples
Uses masked-loss equations, PyTorch reductions, unequal-length loss lists, broadcast sequence advantages, and token-level GAE advantages.
What it assumes
Assumes familiarity with token losses and minibatch optimization while connecting aggregation to advantages, rewards, KL penalties, and RL algorithms.
What you can do after
Apply it via masked tensor reductions while choosing normalization according to sequence weighting, variance, stability, and training setup.
How it connects
Connects sequence length, gradient accumulation, PPO, RLOO, GRPO, KL penalties, MDP framing, and bandit framing.
Cover of Reinforcement Learning from Human Feedback
liveBookmanning.com
Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.