liveBook · Manning

Reinforcement learning with verifiable rewards

As explained in LLM Customization and Fine-Tuning

A method of fine-tuning a model with reinforcement learning using feedback on its outputs.

When a model produces an answer that a program can check, human preference labels may be an unnecessary intermediary. Reinforcement learning with verifiable rewards (RLVR), often called reinforcement fine-tuning (RFT) on managed platforms, uses a programmable grader to score sampled outputs and updates the model to favor higher-scoring responses. section 8.5

The verifier determines what the model is rewarded for. Practical enterprise candidates include:

  • Structured extraction: compare invoice fields with ground truth. section 8.5.1
  • SQL generation: check whether a query returns the correct result set on a test database. section 8.5.1
  • Compliance checking: grade documents against a checklist of required elements. section 8.5.1
  • Code generation or repair: reward solutions that pass unit tests. section 8.5.1

Grading a diagnostic conversation

AMIE, a diagnostic dialogue system, combined supervised fine-tuning on physician-patient dialogues with self-play against a simulated patient. An automated critic evaluated diagnostic questions, differential diagnoses, and next steps. An inner loop refined questioning, while an outer loop folded the best simulated conversations back into later fine-tuning. This illustrates a programmatically scored multi-step process rather than a single final answer. section 8.5.1

RLVR can be data-efficient because the verifier can score many generated outputs. Reported studies found that training with one correct example could nearly double performance on MATH-500 relative to the base model. Other findings suggest that checking only final answers can incentivize more structured reasoning; this is an observed effect, not a guarantee for every task. section 8.5.3

Choose RLVR when outcomes are programmatically checkable, the quality gap is substantial, and the compute budget supports extended training. A planning estimate puts training at 100–700 times the compute of a standard fine-tune; generating multiple responses per prompt is a major cost. For tasks without verifiable outcomes, such as open-ended advice, consider preference pairs with DPO or a trained reward model with RLHF instead. Check training-data quality first, and run safety regression tests after preference optimization. section 8.5.5 section 8.8.1 section 8.10

How LLM Customization and Fine-Tuning covers it

How deep it goes
Explains the sampling, grading, and update loop, then compares applicability and estimated compute costs without specifying an implementation.
Examples
Illustrates grading with invoice extraction, SQL result sets, compliance checklists, unit tests, and simulated diagnostic conversations.
What you can do after
Evaluate whether a task's verifier and quality gap justify reinforcement fine-tuning's compute cost.
How it connects
Connects verifiable rewards with preference optimization, GRPO, supervised fine-tuning, self-play, DPO, and RLHF.

Also explained in

Compare how 6 books explain reinforcement learning with verifiable rewards →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.