liveBook · Manning

Proximal Policy Optimization

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Proximal policy optimization is a policy-gradient reinforcement-learning method that clips policy updates to keep the updated policy close to the previous one.

Proximal Policy Optimization (PPO) is a policy-gradient method for updating a policy toward responses with above-average reward while clipping the update size. The policy therefore stays close to its previous version, trading slower progress for greater training stability. section 7.5.4

A practical PPO update follows this pattern:

  • Estimate which sampled responses receive higher reward than average. section 7.5.4
  • Increase the probability of those responses through a gradient step. section 7.5.4
  • Clip the step so an unusually high-reward response cannot move the policy too far in one update. section 7.5.4
  • In RLHF, subtract a KL penalty for divergence from the supervised-fine-tuned reference policy, balancing reward-model gain against behavioral drift. section 7.5.4

PPO in RLHF

The InstructGPT pipeline uses supervised fine-tuning, reward-model training on preference rankings, and PPO with a KL penalty against the supervised-fine-tuned model. This limits drift because the reward model was trained on outputs resembling that starting policy; without such control, unstable updates can produce degenerate outputs or encourage reward hacking. section 7.8 section 7.5.4

PPO and KL control improve stability but do not eliminate reward hacking. The method is common in large RLHF runs, while DPO and GRPO provide alternative optimization approaches; detailed reinforcement-learning training procedures remain outside this treatment. section 7.5.6 section 7.8

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines clipped updates and KL regularization but stops short of detailed reinforcement-learning training procedures.
Examples
Uses the InstructGPT RLHF pipeline and contrasts PPO with DPO and GRPO as alternative policy optimizers.
What it assumes
Assumes basic policy-gradient, reward-model, KL-divergence, and RLHF terminology while leaving training mechanics outside scope.
What you can do after
Evaluate PPO’s role in an RLHF pipeline and diagnose policy drift, instability, and reward-hacking risks.
How it connects
Connects policy gradients, KL divergence, reward models, supervised fine-tuning, RLHF, reward hacking, DPO, and GRPO.

Also explained in

Compare how 6 books explain Proximal Policy Optimization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.