How deeply does Reinforcement Learning from Human Feedback cover clipped surrogate objective?

Derives the ratio-and-advantage objective and explains trust-region, positive-, and negative-advantage cases with per-token loss code.

Clipped surrogate objective

As explained in Reinforcement Learning from Human Feedback

A PPO surrogate objective, also called a clipped surrogate objective, is an importance-weighted policy-gradient objective whose probability ratio is clipped to limit policy updates.

The clipped surrogate objective is PPO’s importance-weighted policy-gradient objective. It reuses data sampled by an old policy while optimizing a current policy, but limits how far the current policy can move in one update. This trades unconstrained improvement for more stable updates. section 6.2.5

Equation 6.28

\[\label{eq:PPO_EQN} J(\theta) = \min\left(\frac{\pi_\theta(a|s)}{\pi_{\theta_{\text{old}}}(a|s)}A, \text{clip} \left( \frac{\pi_\theta(a|s)}{\pi_{\theta_{\text{old}}}(a|s)}, 1-\varepsilon, 1+\varepsilon \right) A \right).\]

For an action a in state s, the ratio compares its probability under the current policy with its probability under the data-collection policy. The objective takes the smaller of the ordinary advantage-weighted ratio and its clipped counterpart. section 6.2.5

Its terms have distinct operational roles:

The policy ratio ρ measures whether the new policy makes a sampled action more or less likely than the old policy. section 6.2.5
The advantage A marks whether the sampled action was better or worse than the value-function baseline. section 6.2.5
The clip range [1−ε, 1+ε] bounds the ratio; outside that range, the objective becomes flat in the direction of an overly large beneficial update. section 6.2.5 section 6.2.6
The minimum handles advantage signs so that positive-advantage actions cannot be made too likely and negative-advantage actions cannot be suppressed too aggressively. section 6.2.6 section 6.3.5

Per-token PPO loss

For language-model RLHF, compute ratio = exp(new_logp − old_logp) per completion token. Form the ordinary and clamped advantage-weighted negative losses, then take their maximum: minimizing that pessimistic loss selects the smaller policy update. Advantages are normalized, and a completion mask excludes padding; a value loss is combined separately. section 6.3.5

Within the trust region, PPO follows the ordinary policy-gradient objective. Clipping changes behavior only when the ratio would move beyond 1±ε, avoiding an explicit trust-region computation while bounding drift from the policy that generated the batch. section 6.2.6 section 6.3.5

The clipped objective is PPO’s central policy mechanism, but practical PPO also uses a learned value function to estimate per-token advantages through generalized advantage estimation. That added component improves the learning pipeline at the cost of memory and implementation complexity. section 6.6

How Reinforcement Learning from Human Feedback covers it

Cover of Reinforcement Learning from Human Feedback

liveBook manning.com

Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Optimization Algorithms: AI techniques for design, planning, and control problemsA clipped objective function restricts the contribution of an optimization objective to a bounded range of values.

Compare how 2 books explain clipped surrogate objective →

Related concepts

Proximal Policy Optimizationused by · Proximal Policy Optimization is a policy-gradient reinforcement-learning algorithm that limits policy updates to keep successive policies close and promote stable learning.
importance weightuses · An importance weight is the ratio of an outcome’s probability under a target distribution or policy to its probability under a sampling or reference distribution or policy, used to reweight that outcome.
policy gradient methodimplements · A policy gradient method is a reinforcement-learning technique that uses gradients of expected return to optimize the parameters of a policy.
objective functionequivalent to · An objective function assigns values to candidate solutions, predictions, or parameter settings according to a criterion to be minimized or maximized.
advantage functionuses · An advantage function in reinforcement learning estimates how much better or worse an action’s expected return is than a baseline value for the corresponding state or policy.

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.