Advantage normalization rescales a batch of advantage estimates to have zero mean and unit variance. It helps keep policy-gradient updates consistent when advantage scales vary during training. section 9.4
In the training flow, normalization follows advantage calculation and precedes the actor update:
- Compute each advantage as return-to-go minus the critic’s value prediction. section 9.4
- Normalize the advantages across the batch, then update the actor using them. section 9.4
