liveBook · Manning

Reinforcement learning with verifiable rewards

As explained in Build a DeepSeek Model (From Scratch)

Reinforcement learning with verifiable rewards is reinforcement learning in which a deterministic procedure evaluates outputs and supplies the reward.

Reinforcement learning with verifiable rewards (RLVR) uses a deterministic checker instead of a learned reward model. The checker evaluates outputs against objective criteria, such as a known mathematical answer, unit tests, a compiler, or a formal rule. section 7.5

A reliable RLVR pipeline must treat verification as part of the training problem:

  • Define a strict response protocol, such as placing the answer after “Final answer:”, so the verifier extracts the intended result rather than prose or multiple candidates. section 7.10.1
  • Extract runnable code deterministically by identifying the expected file or function before executing tests; brittle extraction can create misleading rewards. section 7.10.1
  • Monitor pass rate, response length, KL divergence, invalid-format rate, duplicate outputs, reward variance, and held-out performance rather than average reward alone. section 7.10.2

Match the checker to the task

For an arithmetic problem, compare the final number with the known answer. For a programming problem, run unit tests or a compiler. For helpfulness, harmlessness, creative writing, or nuanced dialogue, use preference data, human evaluation, or a learned reward model because no single deterministic answer exists. section 7.5 section 7.7.5

Figure 7.10 RLVR can replace learned reward models for objectively verifiable tasks such as math and code; open-ended assistant behavior still requires preference signals.

Use the figure to distinguish tasks where exact checkers can replace learned reward models from open-ended behavior that still requires preference signals. section 7.5

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains verifier selection, output parsing, monitoring, and scope limits, but stops short of a complete training implementation.
Examples
Uses arithmetic answers, unit tests, compilers, formal rules, structured final-answer markers, and held-out evaluation sets.
What it assumes
Assumes reinforcement-learning concepts and teaches how objective answer checking changes reward construction and evaluation.
What you can do after
Apply deterministic checkers where exact correctness is available and combine them with preference signals for non-verifiable behavior.
How it connects
Connects deterministic checkers with reward models, preference data, human evaluation, GRPO, and hybrid reward design.

Also explained in

Compare how 6 books explain reinforcement learning with verifiable rewards →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.