Chapter 7: Improving GRPO for reinforcement learning

Introduction

The chapter picks up from an implemented GRPO baseline and identifies training instability as a concern during longer runs.

• Basic GRPO training instability: Explain that basic GRPO can become unstable during longer training even when its implementation is correct.
  Why it matters: This is why a working GRPO implementation still needs attention during longer training runs.

Key terms: GRPO, training instability

7.1 Improving GRPO

This section introduces the chapter’s GRPO experiments and cautions against drawing strong conclusions from individual training runs.

• Interpreting GRPO experiment results: Explain why results from a single GRPO experiment should be treated cautiously and what would be needed to judge the effect of an individual setting more confidently.
  Why it matters: This distinction helps you learn from the training examples without treating one run as definitive evidence.

Key terms: sampling, optimization, training runs, tradeoffs

7.2 Tracking GRPO performance metrics

This section introduces a discussion of GRPO training metrics for interpreting and debugging training behavior.

7.2.1 Executing a GRPO training run

This section shows how to obtain and run the chapter 6 GRPO training script, or use its saved logs instead of running a costly training job.

• Baseline GRPO script execution: Describe how to obtain the supplementary script containing the chapter 6 GRPO training code, check that it downloaded, and run it.
  Why it matters: This gives you a runnable baseline without copying the chapter 6 implementation.
• Saved training log options: Distinguish the two supplied log formats and explain why a reader might use them instead of running training.
  Why it matters: You can inspect the run’s results without paying the computational cost of reproducing it.

Key terms: GRPO, supplementary materials, rlvr_grpo_original_no_kl.py, uv run, log files, .txt, .csv

7.2.2 Inspecting the GRPO training run

The section uses plots of four GRPO training metrics to assess a run, identify signs of instability, and explain how to obtain evaluation accuracy.

• Response length and loss trends: Describe the expected trends in average response length and loss, and identify what the plotted run shows.
  Why it matters: These trends help you spot concerning behavior without treating loss as the sole measure of progress.
• Reward growth and saturation: Explain both the desired direction of average reward and why a reward of 1.00 can justify stopping training.
  Why it matters: A perfect sampled reward can be a stopping cue, not a reason to keep training indefinitely.
• External accuracy and instability: Explain how the MATH-500 accuracy trend changes the interpretation of this training run.
  Why it matters: A fixed external evaluation reveals deterioration that changes in the training examples cannot by themselves explain.
• Evaluation accuracy collection options: Describe how evaluation accuracy can be obtained and the trade-off of calculating it during training.
  Why it matters: You can include an external accuracy measure without unnecessarily slowing the training run.

Key terms: average response length, loss, sanity check, average reward, training signal, stopping early, evaluation accuracy, MATH-500, training instability, eval_acc

7.3 Tracking more advanced GRPO performance metrics

Introduces rollout advantages and generated-sequence entropy as additional metrics for examining GRPO training dynamics.

7.3.1 Advantage tracking

This subsection shows how the mean and standard deviation of GRPO advantages help diagnose training, especially when rewards provide no learning signal.

• Advantage mean as sanity check: Explain why the mean of GRPO advantages is tracked and what it is expected to show.
  Why it matters: The mean helps you check the implementation, but it tells you little about the strength of the learning signal.
• Advantage spread and training stability: Describe what different standard deviations of the advantages suggest about the GRPO learning signal.
  Why it matters: The spread of advantages helps reveal whether policy updates have a useful, stable learning signal.
• Identical rewards and learning signal: Explain why identical rewards in a sampled group stop its policy gradient update, and how average reward changes the interpretation.
  Why it matters: A missing learning signal can mean either unproductive examples or that the model has mastered them.

Key terms: advantages, sample mean, standard deviation, gradient signal, normalized advantages, policy gradient update, average reward

7.3.2 Entropy tracking

Explains how to calculate and track answer-token entropy as a diagnostic of uncertainty and exploration during GRPO training.

• Entropy as training diagnostic: Explain what high, moderate, and low entropy say about the model’s next-token probabilities, and why a sudden drop to very low entropy is concerning during training.
  Why it matters: Entropy helps you tell growing confidence apart from a potentially unhealthy loss of exploration.
• Entropy from token probabilities: Describe how entropy uses probabilities and log probabilities across possible next tokens, and how probabilities can be recovered from log probabilities.
  Why it matters: The calculation connects the model’s token scores to an interpretable measure of uncertainty.
• Average answer-token entropy: Distinguish entropy at one generated token position from the average entropy across answer tokens, and describe how the latter is tracked during GRPO training.
  Why it matters: Averaging across answer tokens provides a metric that can be monitored over training rather than inspecting a single token position.

Key terms: entropy, exploration, training collapse, probs, logprobs, torch.softmax(), torch.log_softmax(), torch.exp(), step_entropy, entropy_all_steps

7.3.3 Plotting additional GRPO metrics

This section interprets plotted advantage statistics, entropy, and average reward together to assess a GRPO training run.

• Zero mean advantage check: Explain why the average advantage stays at zero during GRPO training and what a departure from zero would indicate.
  Why it matters: This check helps distinguish a training signal from a possible calculation error.
• Advantage spread and learning signal: Describe how the advantage standard deviation changes in the plotted run and what that says about rollout quality and the learning signal.
  Why it matters: The spread of advantages says more about whether training can continue than their zero mean does.
• Entropy in training context: Interpret the entropy trend in this run and explain why entropy should be considered alongside reward and advantage spread rather than treated alone as evidence of collapse.
  Why it matters: Reading the metrics together avoids mistaking exploratory behavior for failure or overlooking a collapse signal.

Key terms: average advantage, group mean, GRPO-style normalization, advantage standard deviation, rollouts, learning signal, entropy, next-token probabilities, average reward, collapse

7.4 Stabilizing sequence-level GRPO using clipped policy ratios

This section introduces clipped policy ratios as a way to stabilize GRPO training.

• Clipped Ratio Stabilization: Explain why clipped policy ratios are added to the GRPO loss: they help limit overly large model weight updates and make training more stable, especially over longer runs.
  Why it matters: This explains the problem the upcoming change to GRPO is meant to address.

Key terms: clipped policy ratios, GRPO loss, model weight updates

7.4.1 Computing clipped policy ratios

This section compares old and updated sequence log probabilities, clips their policy ratios, and uses those ratios to limit the influence of rollouts on the GRPO loss.

• Old and new policy ratios: Explain what a policy ratio compares and how to interpret it. Describe why the old and new sequence log probabilities must be computed for the same prompt.
  Why it matters: The ratio shows how much an update changed the model’s likelihood for a sampled answer.
• Clipping range and stability: Describe how clip_eps bounds a policy ratio and explain why clipping can help stabilize an update.
  Why it matters: Clipping keeps an extreme change in likelihood from exerting too much influence on an update.
• Clipped policy gradient loss: Distinguish the section’s clipped policy gradient loss from its earlier policy gradient calculation and explain what clipping changes in the example.
  Why it matters: This connects clipped ratios to the training loss they are meant to stabilize.

Key terms: policy ratio, old logprobs, new logprobs, average policy ratio, clipped_ratio, clip_eps, advantage term, clipped policy gradient loss, clipped objective, policy ratios

7.4.2 Training with clipped policy ratios

This section applies clipped policy ratios in a GRPO training run, contrasts a large fixed-rollout approach with a smaller approximation, and reports improved stability in the resulting metrics.

• Fixed Reference Policy Comparison: Explain why a clipped policy ratio update compares the same sampled responses under a fixed reference policy and the changing current model.
  Why it matters: The comparison only makes sense when the old and new scores refer to the same responses.
• Rollout Reuse Training Approaches: Distinguish DeepSeek-R1’s fixed rollout pool and minibatch updates from the smaller training approximation used in this section.
  Why it matters: It explains how the example preserves the fixed-reference comparison while using fewer resources.
• Observed Training Stability: Describe the stability difference shown by the clipped-policy-ratio training run compared with previous runs.
  Why it matters: The plotted metrics provide the section’s evidence that this training change helped stability.

Key terms: policy ratio, old_logps, new_logps, reference policy, current model, rollout pool, minibatches, reference model, eight rollouts, average reward

7.5 Controlling how much the model changes with a KL term

This section explains how a KL term constrains changes to a GRPO-trained model and how that differs from clipped policy ratios.

• KL constraint on model change: Explain what KL measures in this setting and why a KL term is added to GRPO.
  Why it matters: It explains why GRPO constrains the trained model relative to where it started.
• KL and clipping time scales: Distinguish the changes controlled by clipped policy ratios from those controlled by the KL term.
  Why it matters: The distinction shows why clipping and KL can both be useful controls on model change.

Key terms: KL, Kullback–Leibler divergence, current policy, reference policy, regularizer, clipped policy ratios, KL term, training trajectory

7.5.1 Implementing the KL loss term

This subsection explains how a KL loss term is computed and added to GRPO training to limit changes from a reference model.

• KL loss calculation and meaning: Explain that the KL loss term sums differences between corresponding current-model and reference-model log probabilities, and interpret what a small value suggests.
  Why it matters: The value helps you interpret whether training is keeping the model stable or possibly constraining learning too much.
• KL penalty in total loss: Explain why the KL term is added to the policy gradient loss and how kl_coeff changes its effect.
  Why it matters: This explains the tradeoff between pursuing reward and avoiding abrupt changes in model behavior.
• Reference model choice and cost: Distinguish the KL term's reference model from the one used for clipped policy ratios, and describe the resource cost of keeping it.
  Why it matters: Choosing and retaining the right reference makes the KL comparison meaningful and has a practical resource cost.

Key terms: KL loss term, logprobs, reference logprobs, policy gradient loss, total loss, kl_coeff, reference model, clipped policy ratio

7.5.2 Training with a KL loss term

This section examines a GRPO run with a KL loss term, explains why its sampled surrogate can destabilize training, and discusses possible remedies.

• KL training run collapse: Describe the training run’s initial accuracy improvement and subsequent collapse, using the plotted metrics and generated responses as evidence.
  Why it matters: The observed failure motivates examining what the KL term actually optimizes.
• Policy roles and KL direction: Distinguish the current, fixed reference, and old rollout policies, and explain why the section’s KL-style logprob difference is not an exact KL estimate.
  Why it matters: Keeping the three policies distinct is necessary to understand why this penalty can behave unexpectedly.
• Response length sensitivity: Explain how summing answer-token logprobs without length normalization can make the KL-style objective sensitive to response length.
  Why it matters: Length sensitivity offers one explanation for the growing responses in the failed run.
• Zero reward surrogate failure: Trace what happens to the training signal when rewards reach zero and explain why this sampled KL surrogate can then produce increasingly random outputs.
  Why it matters: This explains why a term intended to restrain policy drift did not rescue the collapsed run.
• KL stability options: Describe the remedies discussed for the unstable run and explain how importance weighting addresses old-policy sampling.
  Why it matters: These options connect each suspected failure mode to a possible training change without treating any one fix as guaranteed.

Key terms: evaluation accuracy, average reward, loss, current trainable policy, reference policy, old rollout policy, sampled surrogate, new_logps, ref_logps, summed logprobs

7.6 Adding an explicit format reward

This section introduces format rewards as optional additions to the answer-correctness reward used in RLVR training.

• Optional format rewards: Explain that an answer-correctness reward is sufficient for RLVR training, but auxiliary rewards can also check whether a generated answer follows formatting guidelines.
  Why it matters: This distinction explains why a training setup might add a format reward even when it already rewards correct answers.

Key terms: answer-correctness reward, auxiliary rewards, format reward, RLVR

7.6.1 Using <think> tokens

The section explains how <think> tags separate reasoning from a final answer, how to make a tokenizer recognize those tags, and how to reward their correct use during GRPO training.

• Reasoning and answer boundaries: Explain how <think> and </think> mark intermediate reasoning while the final answer appears outside the tags, and why this optional format is useful.
  Why it matters: The tags give the model a clear way to distinguish its working from its final answer.
• Tokenizer support for think tags: Describe how encoding reveals whether a tokenizer treats <think> as one token, and how the section obtains tokenizer support for both tags.
  Why it matters: Checking the encoded IDs shows whether the intended tags are recognized as distinct tokens.
• Format reward and training signal: Describe the format reward’s pass condition and how it can be added to GRPO training alongside a prompt that requests the tags.
  Why it matters: A separate format signal can encourage the requested output structure during GRPO training.

Key terms: <think>, </think>, intermediate reasoning, final answer, base tokenizer, reasoning model variant, subword tokens, placeholder token IDs, vocabulary, reward_format

7.6.2 Training a model to emit <think> tokens

This section trains a reasoning-model variant with a <think> format reward and interprets a run in which reward stays high while accuracy declines.

• Model familiarity with think tokens: Explain why this run uses the existing reasoning variant rather than the base model, and how the base model could be prepared to use <think> and </think> tokens.
  Why it matters: A format reward is easier to interpret when the model already recognizes the format being rewarded.
• Format reward accuracy tradeoff: Describe what happened to accuracy and reward during the run, explain the proposed role of the format reward, and state the two suggested ways to change that reward.
  Why it matters: A high total reward can conceal a drop in the answer accuracy the training is meant to improve.
• Zero loss metric interpretation: Explain why the default run logs zero kl_loss and zero scalar loss, and identify the metrics the section recommends monitoring instead.
  Why it matters: It prevents a zero logged loss from being mistaken for proof that the training run has succeeded or stopped changing.

Key terms: base model, reasoning variant, <think> and </think> tokens, instruction fine-tune, eval_acc, reward_avg, format_reward_avg, format_reward_weight, rlvr_reward, kl_loss

7.6.3 More GRPO modifications, tips, and tricks

This section surveys proposed GRPO modifications and briefly explains the training problems each aims to address.

• Informative prompt selection: Explain why GRPO can skip prompts whose sampled responses all receive the same reward and prioritize prompts that yield mixed results.
  Why it matters: Which prompts are sampled affects how much useful feedback a GRPO update receives.
• Loss computation granularity: Distinguish the suggested token-level loss from sequence-level loss, including the stated reason to consider token-level computation.
  Why it matters: The proposals differ in whether they treat tokens or entire responses as the unit of an update.
• Reward and advantage normalization: Describe the contrasting proposals for normalizing advantages and the separate proposal for combining reward components.
  Why it matters: These suggestions change how rewards become comparable learning signals, and they are not all recommendations to remove normalization.
• KL penalty choices: Explain when the listed proposals omit or vary the KL penalty and why one proposal reweights its estimate.
  Why it matters: The section presents KL as a tunable constraint, not a penalty that must always be applied at one fixed strength.
• Off-policy update safeguards: Describe the listed ways to prevent off-policy samples or probability calculations from distorting an update.
  Why it matters: These suggestions target unstable corrections or mismatches between sampled rollouts and the probabilities used for training.

Key terms: zero-gradient examples, active sampling, advantage signal, token-level loss, sequence-level loss, sequence-level importance sampling and clipping, standard-deviation normalization, group-based advantage normalization, reward component, KL loss term

7.7 Summary

The section summarizes how to diagnose GRPO training and how clipping, KL constraints, and format rewards affect stability and response structure.