Gradient descent updates model parameters to improve an optimization objective. In post-training language models, these updates are a central reason reinforcement learning remains useful despite the heavier infrastructure required than instruction tuning or direct preference optimization: they can improve robustness or chat behavior while preserving broad capabilities when applied to a narrow prompt distribution. section 3.1.6
In policy optimization, a gradient step changes the current policy using data generated by an older policy. PPO reuses that batch through an importance ratio: the current policy probability of an action divided by the data-collection policy probability. Because optimizing this ratio without limits can produce destructive policy changes, PPO clips it near 1, constraining how far one update can move the policy. section 6.2.5
Equation 6.28
\[\label{eq:PPO_EQN} J(\theta) = \min\left(\frac{\pi_\theta(a|s)}{\pi_{\theta_{\text{old}}}(a|s)}A, \text{clip} \left( \frac{\pi_\theta(a|s)}{\pi_{\theta_{\text{old}}}(a|s)}, 1-\varepsilon, 1+\varepsilon \right) A \right).\]The clipped surrogate objective selects the more conservative policy-improvement term, limiting the incentive for a gradient step to move the action probability too far from the policy that collected the data. section 6.2.5
One-step PPO or GRPO
Taking exactly one gradient step per sampled completion makes the current and reference policies equal at the update start. The policy-ratio and clipping machinery can then be omitted, yielding an optimizer much closer to standard policy gradient; multiple steps per sample instead require the protective clipping logic. section 6.3.5
With reinforcement learning from verifiable rewards, updates are driven by whether an answer is correct. For code, unit tests can assign reward 1 when all assertions pass and 0 otherwise, or award partial credit; this feedback can train behavior without a learned reward model. section 7.1
