Proximal Policy Optimization (PPO) is a policy-gradient method for updating a policy toward responses with above-average reward while clipping the update size. The policy therefore stays close to its previous version, trading slower progress for greater training stability. section 7.5.4
A practical PPO update follows this pattern:
- Estimate which sampled responses receive higher reward than average. section 7.5.4
- Increase the probability of those responses through a gradient step. section 7.5.4
- Clip the step so an unusually high-reward response cannot move the policy too far in one update. section 7.5.4
- In RLHF, subtract a KL penalty for divergence from the supervised-fine-tuned reference policy, balancing reward-model gain against behavioral drift. section 7.5.4
PPO in RLHF
The InstructGPT pipeline uses supervised fine-tuning, reward-model training on preference rankings, and PPO with a KL penalty against the supervised-fine-tuned model. This limits drift because the reward model was trained on outputs resembling that starting policy; without such control, unstable updates can produce degenerate outputs or encourage reward hacking. section 7.8 section 7.5.4
PPO and KL control improve stability but do not eliminate reward hacking. The method is common in large RLHF runs, while DPO and GRPO provide alternative optimization approaches; detailed reinforcement-learning training procedures remain outside this treatment. section 7.5.6 section 7.8





