A reward model is a learned model that scores generated outputs according to preference-based judgments such as helpfulness, harmlessness, nuance, or style. It supplies a reinforcement-learning signal when the desired behavior cannot be reduced to a deterministic answer. section 7.5.2 section 7.7.5
Reward models complement, rather than eliminate, verifiers. Rule-based checking suits objectively answerable math and code tasks; learned preference signals suit open-ended assistant behavior. section 7.7.1 section 7.7.5
A practical reward-source split is:
- Use deterministic checkers when correctness has an exact, reliable criterion. section 7.12
- Use a reward model when the target depends on human preference, safety, helpfulness, or style. section 7.5.2 section 7.7.5
- Combine both signals when one model must handle verifiable reasoning and general assistant behavior. section 7.7.1
Open-ended helpfulness
A math response can receive a rule-based correctness score, while a helpfulness response may require a learned reward model because no single objectively correct answer exists. section 7.7.5
|
Step
|
Question to ask before scaling
|
|
Task
|
Can the outcome be verified reliably?
|
|
Format
|
Can the final answer be parsed without ambiguity?
|
|
Rollout
|
Are old log probabilities and response masks stored
correctly?
|
|
Reward
|
Are parse failures and invalid outputs tracked separately?
|
|
Advantage
|
Does each prompt group have useful reward variance?
|
|
KL
|
Is the policy staying close enough to the frozen
reference?
|
|
Evaluation
|
Does held-out performance improve, not just training
reward?
|
Use the implementation checklist to decide whether a task needs a verifier or a learned reward model, and to track reward diagnostics during rollout. section 7.12









