liveBook · Manning

Clipped surrogate objective

As explained in Applied Reinforcement Learning

A PPO objective that clips the policy probability ratio to limit the effect of policy updates.

During language-model fine-tuning, PPO uses the clipped surrogate objective to keep policy updates from changing token probabilities too sharply. It compares each token’s probability under the new policy with its probability under the old policy, then limits the ratio’s contribution to the objective. section 10.7

LCLIP(θ) = E[min(rt(θ)Ât, clip(rt(θ), 1 − ε, 1 + ε)Ât)]

The objective uses the smaller of the unclipped and clipped advantage-weighted ratios. section 10.7

Limiting a token update

If the old policy assigns “doctor” probability 0.20 and the new policy assigns 0.24, the ratio is 1.20. With a positive advantage of 2, that contributes 2.40 to the objective. Raising the probability to 0.80 makes the ratio 4.0, but clipping at 1.20 keeps the contribution at 2.40. section 10.7

How Applied Reinforcement Learning covers it

How deep it goes
Works through the ratio, clipping interval, and positive-advantage example without deriving the objective.
Examples
Uses a token probability change from 0.20 to 0.24 and a positive-advantage case where a ratio of 1.80 is clipped to 1.20.
What it assumes
Uses old and new token probabilities and advantage estimates, while explaining how their ratio is clipped.
What you can do after
Understand how PPO clipping limits the objective’s benefit from large policy changes.
How it connects
Connects PPO updates with advantage estimates, policy ratios, trust regions, and reinforcement learning from human feedback.

Also explained in

Compare how 3 books explain clipped surrogate objective →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.