Proximal Policy Optimization (PPO) is an on-policy policy-gradient method for improving a parameterized policy from recently generated rollouts. It reuses a fixed batch for multiple updates while limiting how far the new policy can move from the policy that generated that batch. In RLHF for language models, this turns reward signals into controlled changes to token probabilities. section 6.1 section 6.2.5
For an action or token, define the probability ratio as ρt(θ) = πθ(at|st) / πθold(at|st), where πθ is the policy being optimized and πθold collected the rollout. The ratio corrects the mismatch between current-policy probabilities and old-policy data through importance sampling. PPO maximizes the smaller of the ordinary advantage-weighted term and the same term with ρ clipped to [1−ε, 1+ε]. Thus, its gradient initially matches the ordinary policy gradient, but incentives to make an already-large policy change become flat outside the clipping range. section 6.2.5 section 6.2.6
Equation 6.28
\[\label{eq:PPO_EQN} J(\theta) = \min\left(\frac{\pi_\theta(a|s)}{\pi_{\theta_{\text{old}}}(a|s)}A, \text{clip} \left( \frac{\pi_\theta(a|s)}{\pi_{\theta_{\text{old}}}(a|s)}, 1-\varepsilon, 1+\varepsilon \right) A \right).\]The clipped surrogate objective makes the two competing terms explicit: the unclipped ratio preserves the policy-gradient signal, while the clipped ratio caps a single update’s effective push. section 6.2.5
A language-model PPO update commonly follows this pipeline:
- Generate completions with a rollout policy and retain its per-token log-probabilities as the old policy reference for the batch. section 6.3.5
- Apply a KL penalty to token rewards relative to the training-start reference policy, then accumulate post-KL returns over valid completion tokens. section 6.3.5
- Predict per-token values with a learned value function; use returns minus detached values as advantages, optionally using generalized advantage estimation. section 6.2.7 section 6.6
- Compute new-to-old probability ratios, form clipped and unclipped policy losses, and select the more pessimistic loss so favorable and unfavorable actions cannot be changed too aggressively. section 6.3.5
- Train the value function with its own loss and combine it with the policy loss, masking non-completion tokens and logging update metrics. section 6.2.7 section 6.3.5
Positive and negative advantages
If an action has positive advantage, PPO should increase its probability; once its ratio rises above 1+ε, clipping prevents further incentive for that direction of change. If an action has negative advantage, PPO should decrease its probability; once its ratio falls below 1−ε, clipping similarly prevents excessive correction. Inside this trust region, PPO behaves like an ordinary policy-gradient update. section 6.2.6
PPO’s stability comes with a learned value function, value-loss choices, and added memory and implementation complexity. Compared with REINFORCE and RLOO, it alone requires that learned value function; compared with GRPO, it estimates advantages rather than normalizing rewards across groups. In LM RLHF, clipping and a KL penalty are distinct regularizers: with only one gradient step per rollout batch, PPO’s native clipping may not activate, leaving the KL penalty as the dominant constraint. section 6.2.11 section 6.4.2 section 6.6
