liveBook · Manning

Verifiable reward

As explained in Build a DeepSeek Model (From Scratch)

A verifiable reward is a training reward whose correctness can be determined by an executable checker, test, or other objective procedure.

A verifiable reward assigns training credit when an executable checker, test, or other objective procedure confirms that a generated response is correct. In reinforcement learning, this replaces subjective scoring for tasks such as mathematics and code, allowing rewarded behavior to become more likely. section 7.0 section 7.15

Implementing verifiable rewards requires:

  • Define a task distribution with gold answers, executable tests, or another reliable correctness signal. section 7.12
  • Specify a response protocol, such as placing the final answer after “Final answer:”, so the verifier can parse outputs reliably. section 7.12
  • Record diagnostic outcomes—including parse failure, invalid format, timeout, and unit-test failure—in addition to the reward. section 7.12
  • Normalize rewards across multiple completions for each prompt when using GRPO, while monitoring groups whose rewards are identical. section 7.12 section 7.15

Math and code tasks

A math verifier compares a parsed response with a gold answer; a code verifier executes tests. When correctness cannot be checked objectively—such as open-ended helpfulness, creative writing, subtle factuality, or safety preferences—use a learned reward model or another alignment method instead. section 7.12 section 7.13

Step
Question to ask before scaling
Task
Can the outcome be verified reliably?
Format
Can the final answer be parsed without ambiguity?
Rollout
Are old log probabilities and response masks stored correctly?
Reward
Are parse failures and invalid outputs tracked separately?
Advantage
Does each prompt group have useful reward variance?
KL
Is the policy staying close enough to the frozen reference?
Evaluation
Does held-out performance improve, not just training reward?

Use the checklist table to audit task verifiability, output parsing, rollout storage, reward diagnostics, advantage variance, KL control, and held-out evaluation before scaling. section 7.12

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains checker selection, response protocols, diagnostics, reward variance, and task limits, but stops short of production trainer details.
Examples
Uses gold-answer math checking, executable code tests, parse diagnostics, and a compact GRPO training-step implementation.
What it assumes
Assumes reinforcement-learning basics but grounds reward assignment, verification requirements, and group-relative advantages through implementation steps.
What you can do after
Apply it via executable checkers, structured output protocols, diagnostic logging, and group-relative reward normalization.
How it connects
Connects reinforcement learning, GRPO, PPO, reward models, KL control, and supervised fine-tuning.

Also explained in

Compare how 3 books explain verifiable reward →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.