liveBook · Manning

Loss normalization

As explained in Reinforcement Learning from Human Feedback

Loss normalization is the rescaling of optimization losses or advantage estimates using statistics computed over a specified set of examples.

Loss normalization rescales losses or advantage estimates using statistics from a chosen example set. In reasoning-model reinforcement learning, that set may be a rollout group or an entire batch. section 7.3.3

The main implementation choice is the normalization scope:

  • Group-level normalization follows the original GRPO formulation but may introduce length or difficulty biases through its per-group terms. section 7.3.3
  • Batch-level normalization computes statistics across the batch, providing an alternative used by models such as Magistral and MiMo. section 7.3.3

Choosing the normalization scope

A reasoning-training pipeline can replace GRPO’s per-group normalization with batch-level normalization when length or difficulty differences between groups may distort the optimization signal. section 7.3.3

How Reinforcement Learning from Human Feedback covers it

How deep it goes
Contrasts group-level and batch-level normalization and identifies possible biases, but does not derive their formulas or implementation steps.
Examples
Names Magistral and MiMo as models using batch-level loss or advantage normalization instead of group-level normalization.
What you can do after
Evaluate normalization scope as an optimization trade-off when training reasoning models with group-dependent signals.
How it connects
Connects GRPO, reinforcement learning, advantage estimation, and length or difficulty bias.
Cover of Reinforcement Learning from Human Feedback
liveBookmanning.com
Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.