liveBook · Manning

Reward model

As explained in Build a DeepSeek Model (From Scratch)

A reward model is a learned model that scores outputs according to preference-based or otherwise non-mechanical judgments of desired behavior.

A reward model is a learned model that scores generated outputs according to preference-based judgments such as helpfulness, harmlessness, nuance, or style. It supplies a reinforcement-learning signal when the desired behavior cannot be reduced to a deterministic answer. section 7.5.2 section 7.7.5

Reward models complement, rather than eliminate, verifiers. Rule-based checking suits objectively answerable math and code tasks; learned preference signals suit open-ended assistant behavior. section 7.7.1 section 7.7.5

A practical reward-source split is:

  • Use deterministic checkers when correctness has an exact, reliable criterion. section 7.12
  • Use a reward model when the target depends on human preference, safety, helpfulness, or style. section 7.5.2 section 7.7.5
  • Combine both signals when one model must handle verifiable reasoning and general assistant behavior. section 7.7.1

Open-ended helpfulness

A math response can receive a rule-based correctness score, while a helpfulness response may require a learned reward model because no single objectively correct answer exists. section 7.7.5

Step
Question to ask before scaling
Task
Can the outcome be verified reliably?
Format
Can the final answer be parsed without ambiguity?
Rollout
Are old log probabilities and response masks stored correctly?
Reward
Are parse failures and invalid outputs tracked separately?
Advantage
Does each prompt group have useful reward variance?
KL
Is the policy staying close enough to the frozen reference?
Evaluation
Does held-out performance improve, not just training reward?

Use the implementation checklist to decide whether a task needs a verifier or a learned reward model, and to track reward diagnostics during rollout. section 7.12

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Distinguishes learned preference signals from deterministic verifiers and places them in hybrid reinforcement learning without detailing model internals.
Examples
Uses helpfulness and harmlessness prompts alongside math and code tasks where rule-based checkers provide the alternative.
What it assumes
Assumes reinforcement-learning rewards and verifiers while distinguishing mechanically checkable outcomes from preference-based judgments.
What you can do after
Apply hybrid reward design by using learned preference signals for open-ended tasks and exact checkers for verifiable tasks.
How it connects
Connects reward modeling with reinforcement learning, PPO, GRPO, RLVR, supervised fine-tuning, preference judgments, and verifiers.

Also explained in

Compare how 10 books explain reward model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.