Group Relative Policy Optimization (GRPO) updates a language model from several sampled answers to the same prompt. A verifier scores each complete rollout, and GRPO compares those scores within the group instead of training a separate value model. section 6.2 section 6.13
A practical GRPO update follows this workflow:
- Generate multiple rollouts for one prompt using temperature and top-p sampling. section 6.5
- Assign each complete answer a verifier reward, such as 1.0 only when the mathematical answer is correct and uses the required boxed format. section 6.6
- Normalize each reward against the group mean and standard deviation: advantage equals reward minus group mean, divided by standard deviation plus a small stability constant. section 6.7
- Compute sequence-level log probabilities by summing token log probabilities over each generated answer, then combine them with advantages into the policy-gradient loss. section 6.13 section 6.10
- Backpropagate the GRPO loss in a conventional PyTorch loop, update the model, log metrics, and periodically save checkpoints. section 6.11
Mixed rewards for four rollouts
For rewards [1, 1, 0, 0], normalized advantages are approximately [0.8659, 0.8659, -0.8659, -0.8659]. Positive advantages increase the likelihood of higher-reward answers; negative advantages decrease it. If every rollout receives the same reward, all advantages and the policy-gradient update become zero. section 6.7 section 7.3.1
Figure 6.9 Step-by-step GRPO update for RLVR. (1) A prompt is sampled and multiple rollouts are generated. (2) Each rollout is scored using a verifiable reward. (3) Group-relative advantages are computed from these rewards. (4) The log probability of each rollout under the current model is calculated. (5) Advantages and log probabilities are combined to form the policy gradient loss. (6) A KL regularization term against a reference model is added, and the resulting total loss is used to update the model parameters.
Use the workflow diagram to trace prompt sampling, grouped rollouts, verifier rewards, relative advantages, log probabilities, and the loss update. section 6.2.2
The introductory implementation omits the KL penalty and clipped policy ratios. Later extensions use clipping to limit between-update changes and KL divergence to constrain longer-term drift from a reference model; math-reasoning systems may omit KL when it harms stability or performance. section 7.4 section 7.5





