liveBook · Manning

Proximal Policy Optimization

As explained in Build a DeepSeek Model (From Scratch)

Proximal Policy Optimization is a policy-gradient reinforcement-learning algorithm that constrains policy updates using a clipped or otherwise limited probability ratio.

Proximal Policy Optimization (PPO) stabilizes policy-gradient updates by limiting how far a new policy moves from the policy that generated the sampled actions. It compares their probabilities with a ratio and clips that ratio to prevent overly large updates. section 7.3

PPO-style language-model training commonly uses four components:

  • A trainable policy model that generates responses. section 7.3.1
  • A reward signal from a learned reward model or verifier. section 7.3.1
  • A learned value model that estimates expected reward and provides a baseline for advantage calculation. section 7.3.1
  • A frozen reference model that anchors the policy; a KL penalty discourages excessive drift from it. section 7.3.1
ratioₖ(θ) = πθ(aₖ | sₖ) / πold(aₖ | sₖ)

The ratio compares a sampled action’s probability under the new policy with its probability under the rollout policy. section 7.3

Figure 7.6 PPO-style language-model training uses a trainable policy, a reward signal, a learned value baseline, and a frozen reference policy for KL control.

The diagram shows how the policy, reward, value baseline, and reference policy fit into language-model training. section 7.3.1

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Defines the sampled-action probability ratio and clipping intuition, then identifies PPO’s value baseline and KL reference control.
Examples
Uses sampled language-model tokens to compare new- and old-policy probabilities and outlines four components of training.
What it assumes
Builds the language-generation mapping by treating growing context as state and each sampled token as an action.
What you can do after
Understand how clipping restrains policy updates and distinguish the learned value model from the frozen reference model.
How it connects
Connects policy gradients, trust-region methods, value models, KL control, and Group Relative Policy Optimization.

Also explained in

Compare how 6 books explain Proximal Policy Optimization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.