During language-model fine-tuning, PPO uses the clipped surrogate objective to keep policy updates from changing token probabilities too sharply. It compares each token’s probability under the new policy with its probability under the old policy, then limits the ratio’s contribution to the objective. section 10.7
The objective uses the smaller of the unclipped and clipped advantage-weighted ratios. section 10.7
Limiting a token update
If the old policy assigns “doctor” probability 0.20 and the new policy assigns 0.24, the ratio is 1.20. With a positive advantage of 2, that contributes 2.40 to the objective. Raising the probability to 0.80 makes the ratio 4.0, but clipping at 1.20 keeps the contribution at 2.40. section 10.7


