liveBook · Manning

Advantage normalization

As explained in Applied Reinforcement Learning

Advantage normalization rescales a batch of advantage estimates to have zero mean and unit variance.

Advantage normalization rescales a batch of advantage estimates to have zero mean and unit variance. It helps keep policy-gradient updates consistent when advantage scales vary during training. section 9.4

In the training flow, normalization follows advantage calculation and precedes the actor update:

  • Compute each advantage as return-to-go minus the critic’s value prediction. section 9.4
  • Normalize the advantages across the batch, then update the actor using them. section 9.4

How Applied Reinforcement Learning covers it

How deep it goes
Defines batch normalization by its zero-mean, unit-variance result and practical purpose, without detailing its calculation.
What it assumes
Assumes advantages have been computed from returns-to-go and critic predictions before normalization.
What you can do after
Understand how normalizing advantages helps maintain consistent gradient step scales during training.
How it connects
Connects advantages to critic value predictions, return-to-go, and actor updates in REINFORCE training.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.