liveBook · Manning

Advantage function

As explained in Build a DeepSeek Model (From Scratch)

An advantage is a reinforcement-learning signal representing how much better or worse an action's outcome is than a reference baseline.

An advantage measures how much better or worse an outcome is than a baseline: Aₖ = Rₖ − baseline(sₖ). Positive values favor sampled behavior; negative values push against it. Compared with raw rewards, a baseline can reduce update variance without changing which actions are optimal. section 7.2.3 section 7.1.3

The baseline matters when prompts vary in difficulty. A reward of 1 may be ordinary on an easy prompt, while a reward of 0 can still be comparatively good on a hard one. PPO commonly computes advantages from returns minus learned value estimates; GRPO instead compares rewards within a group of completions for the same prompt. section 7.2.3 section 7.10.3

Aᵢ = (rᵢ - mean(r₁, …, rG)) / (std(r₁, …, rG) + ε)

GRPO subtracts the group mean and divides by the group standard deviation plus ε for numerical stability. section 7.4.1

Binary rewards, signed advantages

Four answers to a math prompt receive rewards [1, 0, 1, 0]. Their mean is 0.5, so the correct answers have positive advantages and the incorrect answers have negative ones. The standard deviation is approximately 0.577 with PyTorch's unbiased default or 0.5 using the population definition; the convention changes the scale, not the signs. section 7.9

Sample multiple completions per prompt, score each one, and normalize rewards within each group. If every reward is identical, the group offers little learning signal; ε prevents division by zero. GRPO eliminates the value-model path, but requires useful reward variation across sampled completions. section 7.10.3 section 7.12 section 7.4.1

A verifier typically scores the whole completion, but the loss uses token log probabilities. The completion-level advantage is broadcast across generated tokens; masks exclude prompt and padding tokens. This is imperfect credit assignment: a failed final answer does not identify which token caused the error. Clipping and reference-policy KL also constrain the resulting update. section 7.9.1 section 7.4.1 section 7.4.2

Listing 7.4 The clipped GRPO loss with reference-policy KL
def grpo_loss(
    logp: torch.Tensor,
    old_logp: torch.Tensor,
    ref_logp: torch.Tensor,
    advantages: torch.Tensor,
    eps: float = 0.2,
    beta: float = 0.04,
) -> torch.Tensor:
    """Compute the clipped GRPO objective as a minimization loss.
 
    Shapes:
        logp, old_logp, ref_logp: [batch_size, group_size, tokens]
        advantages: [batch_size, group_size]
    """
    ratio = torch.exp(logp - old_logp)
    clipped = ratio.clamp(1.0 - eps, 1.0 + eps)
    adv = advantages.unsqueeze(-1)
    surrogate = torch.minimum(ratio * adv, clipped * adv)
    kl = logp - ref_logp
    return -(surrogate - beta * kl).mean()

The PyTorch loss expands advantages from [batch, group] across tokens, combines them with the clipped probability ratio, and adds reference-policy control. This simplified listing omits response and attention masks. section 7.8.3

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Derives group normalization and traces advantages into a clipped loss and training-step skeleton, while deferring production rollout and masking infrastructure.
Examples
Works through four math completions with binary verifier rewards and compares unbiased and population standard deviations.
What you can do after
Apply it in a PyTorch GRPO training-step skeleton by normalizing verifier rewards and passing advantages into a clipped loss.
How it connects
Connects advantages to policy gradients, value functions, PPO, GRPO, sparse rewards, credit assignment, clipping, and KL divergence.

Also explained in

Compare how 5 books explain advantage function →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.