Policy-gradient methods update an LLM from sampled responses: tokens in successful responses become more likely, while tokens in unsuccessful ones become less likely. The method uses the log probabilities of sampled tokens, weighted by reward or advantage. section 7.1.3 section 7.2
A rollout-based update works through the following steps:
- Sample tokens from the model’s next-token distribution; stochastic sampling enables exploration beyond the most likely token. section 7.2.1
- Treat the generated response as a trajectory and use its outcome to weight the sampled tokens’ log probabilities. section 7.2.2
- Apply the same completion-level advantage across its generated tokens; this coarse signal does not identify which token caused success or failure. section 7.9.1
The simplified update makes the central operation explicit: reward scales the log-probability gradient of each sampled token. section 7.2.2
This compact rule omits practical details such as sequence masks, old-policy probabilities, and KL penalties. PPO and GRPO add stabilizing mechanisms; GRPO uses group-relative advantages, clipping, and reference-policy KL. section 7.2.2 section 7.15





