Surrogate objective

Q: How deeply does Reinforcement Learning from Human Feedback cover surrogate objective?

Explains clipping, flat-gradient conditions, clip-fraction monitoring, and the effect of gradient steps per batch.

As explained in Reinforcement Learning from Human Feedback

A surrogate objective is an alternative optimization objective designed to approximate or constrain improvement of a target objective.

A surrogate objective controls PPO update size by clipping the policy probability ratio, while RLHF also regularizes against the original policy with a KL-distance penalty. section 6.4.2

Interpret the clipping behavior operationally:

The clip-fraction variable reports the percentage of batch samples whose probability ratio leaves the clipping interval. section 6.4.2
A sample outside the interval does not always have zero gradient; the surrogate becomes flat only when the clipped branch is selected. section 6.4.2
With one gradient step per batch, PPO clipping may remain inactive, leaving KL penalties as the dominant regularizer; multiple steps can activate clipping. section 6.4.2

Update iterations in practice

Many language-model RLHF runs use one gradient step per batch. DAPO uses 16, while Tülu 3 uses 4 for 8B and 70B models but 1 for 405B to maintain stability. section 6.4.2

How Reinforcement Learning from Human Feedback covers it

Cover of Reinforcement Learning from Human Feedback

liveBook manning.com

Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Related concepts

Proximal Policy Optimizationused by · Proximal Policy Optimization is a policy-gradient reinforcement-learning algorithm that limits policy updates to keep successive policies close and promote stable learning.
surrogate modelcontrasts with · A surrogate model is an approximate model that estimates the behavior or outputs of a more complex, costly, or otherwise unavailable model, process, or function.

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.