How deeply does Reinforcement Learning from Human Feedback cover group-relative advantage normalization?

Defines the algorithmic substitution for PPO’s value function but does not provide equations or implementation steps.

Group-relative advantage normalization

As explained in Reinforcement Learning from Human Feedback

Group-relative advantage normalization computes normalized advantage estimates relative to other sampled outputs in the same group.

Group-relative advantage normalization estimates how good each sampled completion is relative to other completions generated for the same prompt. It replaces PPO’s learned value function with this group-level comparison, reducing the need for value-function training in GRPO. section 6.6

The technique changes the reinforcement-learning pipeline in several practical ways:

Samples multiple completions per prompt so their relative quality can provide the advantage signal. section 6.6
Avoids the learned value function required by PPO, reducing memory and implementation requirements associated with value estimation. section 6.6
Fits value-function-free approaches such as GRPO and RLOO, where advantage estimation differs from PPO’s per-token GAE pipeline. section 6.6

How Reinforcement Learning from Human Feedback covers it

Cover of Reinforcement Learning from Human Feedback

liveBook manning.com

Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Related concepts

Group Relative Policy Optimizationimplemented by · Group Relative Policy Optimization is a reinforcement-learning algorithm that updates a policy using relative rewards or advantages estimated from groups of sampled outputs or actions.
normalizationkind of · Normalization is the process of transforming a value, representation, or data structure to satisfy a specified standard, scale, or structural constraint.
advantage functionmeasures · An advantage function in reinforcement learning estimates how much better or worse an action’s expected return is than a baseline value for the corresponding state or policy.

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.