Proximal Policy Optimization (PPO) stabilizes policy-gradient updates by limiting how far a new policy moves from the policy that generated the sampled actions. It compares their probabilities with a ratio and clips that ratio to prevent overly large updates. section 7.3
PPO-style language-model training commonly uses four components:
- A trainable policy model that generates responses. section 7.3.1
- A reward signal from a learned reward model or verifier. section 7.3.1
- A learned value model that estimates expected reward and provides a baseline for advantage calculation. section 7.3.1
- A frozen reference model that anchors the policy; a KL penalty discourages excessive drift from it. section 7.3.1
The ratio compares a sampled action’s probability under the new policy with its probability under the rollout policy. section 7.3
Figure 7.6 PPO-style language-model training uses a trainable policy, a reward signal, a learned value baseline, and a frozen reference policy for KL control.

The diagram shows how the policy, reward, value baseline, and reference policy fit into language-model training. section 7.3.1





