Reinforcement learning with verifiable rewards

As explained in Build a Reasoning Model (From Scratch)

A reinforcement learning approach that uses automatically checkable correctness rewards instead of human preference signals.

Reinforcement learning with verifiable rewards (RLVR) trains an LLM with automatically checkable correctness signals instead of human preference labels or a learned reward model. A model generates a response, a deterministic verifier assigns a reward, and the training objective updates the model directly. section 6.1.2

For a math task, a verifier compares the generated final answer with ground truth and returns 1 for correct or 0 for incorrect. RLVR therefore collapses the two-stage RLHF pipeline into one generation, verification, and update loop. section 6.1.2

A practical RLVR implementation combines these components:

• Define a task with an automatic correctness checker, such as mathematical answer verification. section 6.1.2
• Sample multiple responses for each prompt and compare their rewards within a group. section 6.0 section 6.2
• Use GRPO to convert the group-relative signal into a loss and update the LLM weights. section 6.2
• Run the resulting loss inside a conventional PyTorch loop with gradient reset, backpropagation, optimizer steps, logging, and checkpoints. section 6.11

Math verification and code execution

A math verifier can label a generated solution correct when its final answer matches ground truth. Code tasks can use compilation and execution as binary checks, but secure sandboxing is required and code execution is not implemented here. section 6.1.2

Figure 6.6 Overview of RLVR. The LLM generates a response that is evaluated by a deterministic verifier, such as the math verifier from chapter 3, which assigns a correctness label that’s used as a reward signal within an RL objective to update the model.

Use the workflow diagram to trace generation, deterministic verification, reward assignment, and policy updating. section 6.1.2

How Build a Reasoning Model (From Scratch) covers it

How deep it goes

Explains deterministic rewards, group sampling, GRPO loss, and the full training loop, but defers secure code-execution environments.

Examples

Uses binary math-answer verification, code compilation or execution, sampled response groups, and a PyTorch training loop with checkpoints.

What it assumes

Assumes familiarity with LLM training loops and introduces RL-specific policy terminology alongside verifiers and group-relative optimization.

What you can do after

Build it from scratch with a verifier, sampled rollouts, GRPO loss, optimizer updates, metrics, and model checkpoints.

How it connects

Connects reinforcement learning from human feedback, reward models, policy optimization, GRPO, PPO, and mathematical verification.

liveBookmanning.com

Read Build a Reasoning Model (From Scratch) on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Reinforcement Learning from Human FeedbackReinforcement learning with verifiable rewards (RLVR) is reinforcement learning in which rewards are derived from objectively checkable outcomes.

Compare how 2 books explain reinforcement learning with verifiable rewards →

Related concepts

• reward modelcontrasts with · A reward model is a learned model that assigns scores to outputs, actions, or outcomes to represent their desirability according to preferences or evaluation criteria.
• verifiable rewarduses · A verifiable reward is a reward signal whose correctness can be determined by an objective criterion or procedure.
• reinforcement learningkind of · Reinforcement learning is a machine-learning paradigm in which an agent learns actions or a policy through interaction with an environment and evaluative feedback, such as rewards.
• verifieruses · A verifier is a person, component, or procedure that checks whether a claim, result, artifact, or credential satisfies specified correctness or validity criteria.
• reinforcement learning with human feedbackcontrasts with · Reinforcement learning with human feedback is a model-training approach that uses human evaluations or preferences to guide optimization of model behavior.

This explainer was generated by Manning's knowledge graph from the text of Build a Reasoning Model (From Scratch). The section links point to the parts of the book it draws on.