A reward signal is a numerical measure of how desirable an agent’s action or completed response is. In language-model post-training, a verifier can turn a completed response into a scalar reward. The policy update uses that feedback to make rewarded behavior more likely, without requiring a target for every intermediate token. section 7.1 section 7.11
During generation, the state is the prompt plus the tokens generated so far, and each sampled next token is an action. The completed response is a trajectory; its final reward supplies feedback for the sampled tokens along that trajectory. Sampling allows the policy to explore different responses to the same prompt. section 7.1.2 section 7.2.1
A basic policy-gradient update multiplies the gradient of each sampled token’s log probability by the response reward. Raw reward is a blunt learning signal, however: a score of 1 on an easy prompt may be less informative than the same score on a hard prompt. An advantage compares reward with a baseline, giving a relative signal that can reduce update variance. section 7.2.2 section 7.2.3
GRPO computes each completion’s advantage from its reward relative to the mean and standard deviation of rewards for the same prompt. The stability constant prevents division by zero when rewards are identical. section 7.4.1
Four answers, one prompt
For a problem with the correct answer 11, four sampled completions receive rewards [1, 0, 1, 0]. Their mean reward is 0.5, so the correct completions have positive group-relative advantages and the incorrect ones have negative advantages. The update favors the former and discourages the latter, subject to clipping and KL control. The verifier identifies failed final answers, not the specific reasoning mistakes behind them. section 7.9
Reward design shapes the behavior being optimized. A math verifier that checks only the last number may reward a correct result with messy reasoning; a format check can encourage a distinct final-answer region. Extraction must also tolerate harmless variation: a brittle parser can penalize a correct answer or assign a misleading reward. section 7.5.1 section 7.10.1
Listing 7.5 One GRPO + RLVR training-step skeleton
def train_step(batch, policy, reference, optimizer, group_size: int = 4):
"""One GRPO + RLVR training step skeleton."""
prompts, gold_answers = batch
completions, old_logp = sample_group(policy, prompts, group_size)
rewards = torch.tensor(
[
[verify_math_answer(y, gold) for y in group]
for group, gold in zip(completions, gold_answers)
],
device=old_logp.device,
)
advantages = group_advantages(rewards)
logp = sequence_logprobs(policy, prompts, completions)
with torch.no_grad():
ref_logp = sequence_logprobs(reference, prompts, completions)
loss = grpo_loss(logp, old_logp, ref_logp, advantages)
optimizer.zero_grad(set_to_none=True)
loss.backward()
optimizer.step()
return {"loss": float(loss.detach()), "reward": float(rewards.mean())}The training-step skeleton shows where rewards enter: sample completions, verify them, compute group advantages, then use those advantages with policy log probabilities in the GRPO loss. Sampling and sequence scoring remain integration placeholders. section 7.8.4















