liveBook · Manning

Reward signal

As explained in Build a DeepSeek Model (From Scratch)

A scalar reward is a single numerical value representing the quality or desirability of an agent's completed response or action.

A reward signal is a numerical measure of how desirable an agent’s action or completed response is. In language-model post-training, a verifier can turn a completed response into a scalar reward. The policy update uses that feedback to make rewarded behavior more likely, without requiring a target for every intermediate token. section 7.1 section 7.11

During generation, the state is the prompt plus the tokens generated so far, and each sampled next token is an action. The completed response is a trajectory; its final reward supplies feedback for the sampled tokens along that trajectory. Sampling allows the policy to explore different responses to the same prompt. section 7.1.2 section 7.2.1

A basic policy-gradient update multiplies the gradient of each sampled token’s log probability by the response reward. Raw reward is a blunt learning signal, however: a score of 1 on an easy prompt may be less informative than the same score on a hard prompt. An advantage compares reward with a baseline, giving a relative signal that can reduce update variance. section 7.2.2 section 7.2.3

Aᵢ = (rᵢ - mean(r₁, …, rG)) / (std(r₁, …, rG) + ε)

GRPO computes each completion’s advantage from its reward relative to the mean and standard deviation of rewards for the same prompt. The stability constant prevents division by zero when rewards are identical. section 7.4.1

Four answers, one prompt

For a problem with the correct answer 11, four sampled completions receive rewards [1, 0, 1, 0]. Their mean reward is 0.5, so the correct completions have positive group-relative advantages and the incorrect ones have negative advantages. The update favors the former and discourages the latter, subject to clipping and KL control. The verifier identifies failed final answers, not the specific reasoning mistakes behind them. section 7.9

Reward design shapes the behavior being optimized. A math verifier that checks only the last number may reward a correct result with messy reasoning; a format check can encourage a distinct final-answer region. Extraction must also tolerate harmless variation: a brittle parser can penalize a correct answer or assign a misleading reward. section 7.5.1 section 7.10.1

Listing 7.5 One GRPO + RLVR training-step skeleton
def train_step(batch, policy, reference, optimizer, group_size: int = 4):
    """One GRPO + RLVR training step skeleton."""
    prompts, gold_answers = batch
    completions, old_logp = sample_group(policy, prompts, group_size)
 
    rewards = torch.tensor(
        [
            [verify_math_answer(y, gold) for y in group]
            for group, gold in zip(completions, gold_answers)
        ],
        device=old_logp.device,
    )
    advantages = group_advantages(rewards)
 
    logp = sequence_logprobs(policy, prompts, completions)
    with torch.no_grad():
        ref_logp = sequence_logprobs(reference, prompts, completions)
 
    loss = grpo_loss(logp, old_logp, ref_logp, advantages)
    optimizer.zero_grad(set_to_none=True)
    loss.backward()
    optimizer.step()
    return {"loss": float(loss.detach()), "reward": float(rewards.mean())}

The training-step skeleton shows where rewards enter: sample completions, verify them, compute group advantages, then use those advantages with policy log probabilities in the GRPO loss. Sampling and sequence scoring remain integration placeholders. section 7.8.4

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Traces rewards from verification through advantage calculation and a GRPO training skeleton, leaving rollout and sequence scoring as integration placeholders.
Examples
Works through four arithmetic completions with binary rewards and group-relative advantages.
What you can do after
Evaluate reward signals by tracing verifier output into relative advantages and considering whether answer extraction reliably measures correctness.
How it connects
Connects reward signals to policy gradients, baselines, advantages, verifiers, PPO, and GRPO.

Also explained in

Compare how 16 books explain reward signal →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.