liveBook · Manning

Reinforcement learning with verifiable rewards

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Reinforcement learning from verifiable rewards is a training method that optimizes a model using rewards computed from objectively verifiable outcomes.

Reinforcement learning with verifiable rewards (RLVR) optimizes a model with rewards computed by deterministic checks rather than a learned preference model. It targets reward hacking: models can exploit approximate reward models while producing nonsensical outputs. section 2.3.5 section 7.7.3

Table 7.1 Task types mapped to reward types
Chapter 1 task type
Chapter 7 reward type
Verifiable task
Programmatic verifier
Open-ended task
Preference-trained reward model

Classify the task before selecting the reward: use a programmatic verifier when ground truth is objective and unambiguous; use preference-trained rewards for open-ended judgment. section 7.1.1

Apply RLVR through a short diagnostic procedure:

  • Identify whether the task has programmatic ground truth and whether that ground truth captures the full objective. section 7.1.1
  • Implement a verifier for exact answers, executable behavior, or structural constraints. section 7.7.3
  • Return a binary reward—1 for passing and 0 for failing—and optimize the policy to produce more passing outputs. section 7.1.1 section 7.7.3
  • Use preference-based signals alongside RLVR when helpfulness, faithfulness, safety, or tone cannot be fully verified. section 7.7.3

Financial research assistant

Verify that a response cites its retrieved source, conforms to the required JSON schema, and reproduces a revenue figure from the transcript. These checks convert otherwise manual judgments into verifiable reward components, while broader helpfulness and harmlessness remain preference-based. section 7.7.3

Lexical anchoring

For instruction following, BLEU can compare a student response with a high-quality synthetic reference. N-gram overlap rewards reproduction of expected facts and formatting, reducing hallucination and semantic drift, but lexical metrics remain lossy proxies for open-ended quality. section 2.3.5 section 7.1.1

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains task classification, verifier construction, binary rewards, and hybrid signals, while deferring open-ended judgment to preference methods.
Examples
Uses exact math answers, unit-tested code, regex-constrained outputs, financial citations, JSON schemas, and BLEU-matched references.
What you can do after
Apply it via exact-answer comparisons, unit-test suites, regex checks, and mixed preference rewards for unverifiable objectives.
How it connects
Connects programmatic verifiers with reward hacking, RLHF, process supervision, reward models, and preference-based training.

Also explained in

Compare how 6 books explain reinforcement learning with verifiable rewards →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.