liveBook · Manning

Group Relative Policy Optimization

As explained in Build a Reasoning Model (From Scratch)

A reinforcement-learning method that uses groups of rollouts for the same prompt and derives learning signals from their relative performance.

Group Relative Policy Optimization (GRPO) updates a language model from several sampled answers to the same prompt. A verifier scores each complete rollout, and GRPO compares those scores within the group instead of training a separate value model. section 6.2 section 6.13

A practical GRPO update follows this workflow:

  • Generate multiple rollouts for one prompt using temperature and top-p sampling. section 6.5
  • Assign each complete answer a verifier reward, such as 1.0 only when the mathematical answer is correct and uses the required boxed format. section 6.6
  • Normalize each reward against the group mean and standard deviation: advantage equals reward minus group mean, divided by standard deviation plus a small stability constant. section 6.7
  • Compute sequence-level log probabilities by summing token log probabilities over each generated answer, then combine them with advantages into the policy-gradient loss. section 6.13 section 6.10
  • Backpropagate the GRPO loss in a conventional PyTorch loop, update the model, log metrics, and periodically save checkpoints. section 6.11

Mixed rewards for four rollouts

For rewards [1, 1, 0, 0], normalized advantages are approximately [0.8659, 0.8659, -0.8659, -0.8659]. Positive advantages increase the likelihood of higher-reward answers; negative advantages decrease it. If every rollout receives the same reward, all advantages and the policy-gradient update become zero. section 6.7 section 7.3.1

Figure 6.9 Step-by-step GRPO update for RLVR. (1) A prompt is sampled and multiple rollouts are generated. (2) Each rollout is scored using a verifiable reward. (3) Group-relative advantages are computed from these rewards. (4) The log probability of each rollout under the current model is calculated. (5) Advantages and log probabilities are combined to form the policy gradient loss. (6) A KL regularization term against a reference model is added, and the resulting total loss is used to update the model parameters.

Use the workflow diagram to trace prompt sampling, grouped rollouts, verifier rewards, relative advantages, log probabilities, and the loss update. section 6.2.2

The introductory implementation omits the KL penalty and clipped policy ratios. Later extensions use clipping to limit between-update changes and KL divergence to constrain longer-term drift from a reference model; math-reasoning systems may omit KL when it harms stability or performance. section 7.4 section 7.5

How Build a Reasoning Model (From Scratch) covers it

How deep it goes
Explains and implements rollout sampling, verification, advantage normalization, log probabilities, loss computation, and training, then defers extensions to later treatment.
Examples
Demonstrates MATH problems, boxed-answer verification, four-rollout rewards, zero-advantage groups, and a PyTorch RLVR training run.
What it assumes
Assumes a pretrained language model, tokenizer, prompt dataset, verifier, and PyTorch training-loop fundamentals.
What you can do after
Build it from scratch as a PyTorch RLVR training loop with grouped sampling, deterministic rewards, normalized advantages, checkpoints, and diagnostics.
How it connects
Connects RLVR, policy gradients, PPO, REINFORCE, sequence-level loss, clipped policy ratios, and KL divergence.

Also explained in

Compare how 6 books explain Group Relative Policy Optimization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a Reasoning Model (From Scratch). The section links point to the parts of the book it draws on.