liveBook · Manning

PPO clipping

As explained in Applied Reinforcement Learning

PPO clipping restricts the optimization benefit of policy probability ratios that exceed a chosen interval.

PPO clipping limits how much a policy update benefits from changing an action’s probability. The probability ratio compares the new policy with the old one; with ε = 0.2, the objective clips that ratio to 0.8–1.2. This discourages large policy jumps. section 10.7

LCLIP(θ) = E[min(rt(θ)Ât, clip(rt(θ), 1 − ε, 1 + ε)Ât)]

The clipped objective shows how PPO compares the unclipped and clipped ratio terms, using the smaller one. section 10.7

A large probability increase

For a positive advantage of 2, a ratio of 1.2 contributes 2.4 to the objective. A ratio of 1.8 is clipped to 1.2, so it contributes no more. Negative advantages likewise limit how sharply a token’s probability can fall in one update. section 10.7

How Applied Reinforcement Learning covers it

How deep it goes
Explains the clipping intuition and works through ratio examples without deriving the objective.
Examples
Uses probability ratios of 1.1, 1.2, and 1.8, plus a negative-advantage token example.
What it assumes
Uses advantage estimates and explains how their positive or negative values guide policy changes.
What you can do after
Understand how clipping limits the objective’s benefit from large policy-probability changes.
How it connects
Connects probability ratios, advantage estimates, and trust regions to PPO policy updates.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.