An advantage measures how much better or worse an outcome is than a baseline: Aₖ = Rₖ − baseline(sₖ). Positive values favor sampled behavior; negative values push against it. Compared with raw rewards, a baseline can reduce update variance without changing which actions are optimal. section 7.2.3 section 7.1.3
The baseline matters when prompts vary in difficulty. A reward of 1 may be ordinary on an easy prompt, while a reward of 0 can still be comparatively good on a hard one. PPO commonly computes advantages from returns minus learned value estimates; GRPO instead compares rewards within a group of completions for the same prompt. section 7.2.3 section 7.10.3
GRPO subtracts the group mean and divides by the group standard deviation plus ε for numerical stability. section 7.4.1
Binary rewards, signed advantages
Four answers to a math prompt receive rewards [1, 0, 1, 0]. Their mean is 0.5, so the correct answers have positive advantages and the incorrect answers have negative ones. The standard deviation is approximately 0.577 with PyTorch's unbiased default or 0.5 using the population definition; the convention changes the scale, not the signs. section 7.9
Sample multiple completions per prompt, score each one, and normalize rewards within each group. If every reward is identical, the group offers little learning signal; ε prevents division by zero. GRPO eliminates the value-model path, but requires useful reward variation across sampled completions. section 7.10.3 section 7.12 section 7.4.1
A verifier typically scores the whole completion, but the loss uses token log probabilities. The completion-level advantage is broadcast across generated tokens; masks exclude prompt and padding tokens. This is imperfect credit assignment: a failed final answer does not identify which token caused the error. Clipping and reference-policy KL also constrain the resulting update. section 7.9.1 section 7.4.1 section 7.4.2
Listing 7.4 The clipped GRPO loss with reference-policy KL
def grpo_loss(
logp: torch.Tensor,
old_logp: torch.Tensor,
ref_logp: torch.Tensor,
advantages: torch.Tensor,
eps: float = 0.2,
beta: float = 0.04,
) -> torch.Tensor:
"""Compute the clipped GRPO objective as a minimization loss.
Shapes:
logp, old_logp, ref_logp: [batch_size, group_size, tokens]
advantages: [batch_size, group_size]
"""
ratio = torch.exp(logp - old_logp)
clipped = ratio.clamp(1.0 - eps, 1.0 + eps)
adv = advantages.unsqueeze(-1)
surrogate = torch.minimum(ratio * adv, clipped * adv)
kl = logp - ref_logp
return -(surrogate - beta * kl).mean()The PyTorch loss expands advantages from [batch, group] across tokens, combines them with the clipped probability ratio, and adds reference-policy control. This simplified listing omits response and attention masks. section 7.8.3




