liveBook · Manning

Group Relative Policy Optimization

As explained in Build a DeepSeek Model (From Scratch)

GRPO, or Group Relative Policy Optimization, is a reinforcement-learning algorithm that estimates relative advantages among grouped sampled responses and optimizes a policy with a clipped objective and regularization.

Group Relative Policy Optimization (GRPO) is a reinforcement-learning method for improving a language model from rewards on sampled completions. It replaces PPO’s learned value-model baseline with a comparison among completions generated for the same prompt. This removes the value-model training path, but retains a frozen reference policy for KL control. section 7.4 section 7.3.3

For each prompt, sample a group of completions, score each one, and normalize its rewards within the group: advantage = (reward − group mean) / group standard deviation. This produces a prompt-specific learning signal without a critic. Multiple completions are essential; identical rewards provide little group-relative signal. section 7.8.2 section 7.12 section 7.10.3

Four completions, one prompt

For a question whose correct answer is 11, suppose four completions receive rewards [1, 0, 1, 0]. The mean is 0.5: the correct completions have positive advantages and the incorrect ones negative advantages. With PyTorch’s unbiased standard deviation, the standard deviation is approximately 0.577. The verifier identifies failed answers, not the individual reasoning mistakes behind them. section 7.9

Listing 7.3 Computing group-relative advantages
import torch
def group_advantages(rewards: torch.Tensor, eps: float = 1e-6) -> torch.Tensor:
    """Normalize rewards within each prompt group.
 
    Args:
        rewards: Tensor of shape [batch_size, group_size].
        eps: Minimum standard deviation for numerical stability.
 
    Returns:
        Tensor of normalized advantages with the same shape as rewards.
    """
    mean = rewards.mean(dim=1, keepdim=True)
    std = rewards.std(dim=1, keepdim=True).clamp_min(eps)
    return (rewards - mean) / std

The PyTorch function normalizes a [batch_size, group_size] reward tensor along the group dimension and clamps the standard deviation for numerical stability. section 7.8.2

The update compares current-policy probabilities with the old-policy probabilities stored during sampling. A clipped surrogate limits update pressure, while a separate KL penalty discourages drift from the frozen reference. For an optimizer that minimizes loss, negate the objective. section 7.4.2 section 7.4

LGRPO(θ) = E[min(ratioₖ(θ)Aᵢ, clip(ratioₖ(θ), 1-ε, 1+ε)Aᵢ) - β KL(πθ || πref)]

The objective combines group-relative advantages with the clipped probability ratio and reference-policy KL penalty. section 7.4.2

A practical training step follows this order:

  • Sample multiple completions per prompt and retain their old-policy token log probabilities. section 7.8.4
  • Score completions with a verifier or reward model, then normalize rewards within each prompt group. section 7.8.4 section 7.4.3
  • Score the generated tokens under the current and frozen reference policies, compute the clipped loss with KL control, and update the policy. section 7.8.4 section 7.4.2
  • Mask prompt and padding tokens out of the policy loss. A simplified log-probability difference is not the preferred production KL estimate; DeepSeek-style GRPO uses the low-variance k3 estimator. section 7.8.3

GRPO saves the value-model forward and training path, not rollout sampling: a fair cost comparison with PPO holds the total number of sampled completions fixed. Larger groups improve the prompt-specific comparison but consume more rollout compute and leave room for fewer distinct prompts. section 7.4.3 section 7.9.2

Monitor more than mean reward: pass rate, response length, reference KL, format failures, duplicate outputs, and within-group reward variance reveal different failure modes. Verify progress on held-out tasks; a longer response is useful only if correctness improves. Verifiable rewards work for objectively checkable tasks, but do not by themselves solve open-ended alignment. section 7.10.2 section 7.13

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Develops reward normalization, a clipped objective, and a training-step skeleton, while deferring distributed training and production infrastructure.
Examples
Works through four math completions with binary rewards and implements group normalization and a clipped loss in PyTorch.
What you can do after
Build a minimal GRPO training-step skeleton from scratch, with placeholders for rollout generation and token log-probability scoring.
How it connects
Connects policy gradients, PPO, value functions, KL divergence, and reinforcement learning with verifiable rewards.

Also explained in

Compare how 6 books explain Group Relative Policy Optimization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.