Verifiable reward

Q: How deeply does Reinforcement Learning from Human Feedback cover verifiable reward?

Defines verifier-scored rewards and illustrates extraction plus binary scoring, while deferring detailed optimizer implementation.

As explained in Reinforcement Learning from Human Feedback

A verifiable reward is a reward signal whose correctness can be checked objectively by a procedure or criterion.

A verifiable reward is an objective training signal whose correctness is determined by a scoring procedure rather than a learned judgment. In reinforcement learning from verifiable rewards (RLVR), the procedure can assign a positive reward to a correct completion and 0 to an incorrect one, making a reward model optional. section 7.1

Verifiable rewards fit tasks with definitive outcomes, such as mathematics, coding, or theorem proving. They contrast with preference-based rewards: judging an explanation of opportunity cost requires assessments of clarity, completeness, and helpfulness, while checking a final mathematical answer can be deterministic. section 7.1 section 7.3.1

A practical verification pipeline typically needs:

A task with a checkable target, such as a correct answer, executable program behavior, or proved statement. section 7.3.1
An answer-extraction convention that isolates the result to be checked. section 7.1
A verifier or scoring function that maps the extracted result to reward, often binary correctness. section 7.1

Checking a mathematical completion

For the sum of primes below 20, a completion may place 77 in a boxed final-answer field. A simple extraction rule reads that field; if the extracted answer equals 77, the verifier returns reward 1. The verifier evaluates the answer outcome, not whether every written reasoning step is stylistically preferred. section 7.1

RLVR uses policy-gradient training much like RLHF, but replaces a learned reward model with a verifier where possible. This reduces dependence on proxy preference signals for checkable tasks, while limiting direct applicability to domains where correctness can actually be scored objectively. section 6.6 section 7.1

Modern reasoning training applies this setup at scale on pretrained models. Earlier approaches used ground-truth signals, binary math rewards, code execution, and theorem verification; accessible tooling now includes TRL, Open Instruct, veRL, and OpenRLHF. section 7.2.1 section 7.3.1

How Reinforcement Learning from Human Feedback covers it

Cover of Reinforcement Learning from Human Feedback

liveBook manning.com

Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Build a Reasoning Model (From Scratch)A reward signal that can be automatically checked against a correct answer or objective criterion.

Compare how 2 books explain verifiable reward →

Related concepts

reinforcement learning with verifiable rewardsused by · Reinforcement learning with verifiable rewards is a training approach that optimizes a model using rewards derived from objectively checkable outcomes.
Group Relative Policy Optimizationused by · Group Relative Policy Optimization is a reinforcement-learning algorithm that updates a policy using relative rewards or advantages estimated from groups of sampled outputs or actions.
reward hackingcontrasts with · Reward hacking is behavior that exploits weaknesses or mismatches in a reward or evaluation specification to obtain a high measured score without achieving the intended objective.
reinforcement learning with human feedbackcontrasts with · Reinforcement learning with human feedback is a model-training approach that uses human evaluations or preferences to guide optimization of model behavior.
reinforcement learningused by · Reinforcement learning is a machine-learning paradigm in which an agent learns actions or a policy through interaction with an environment and evaluative feedback, such as rewards.

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.