liveBook · Manning

Verifier

As explained in LLM Evaluation and Alignment, The Foundational Ideas

A verifier is a deterministic or objective computational system that checks whether an output satisfies a correctness criterion.

A verifier is a deterministic or objective computational system that checks whether an output satisfies a correctness criterion. It returns a passing or failing signal, often 1 or 0, that can measure an answer or serve as a reinforcement-learning reward. section 7.1.1 section 7.7.3

Use a verifier when the task has a correct outcome, correctness can be checked programmatically, and the criteria are objective and unambiguous. If any condition fails—as with tone, helpfulness, or open-ended faithfulness—a fixed verifier score can reward shortcuts rather than task success. section 7.1.1 section 7.7.3

Design a verifier by matching the check to the output:

  • Compare a mathematical final answer with ground truth. section 7.7.3
  • Execute generated code against a unit-test suite. section 7.7.3
  • Validate structured output with a regular expression or schema-like format check. section 7.7.3 section 7.1.1
  • Reserve preference-based signals for criteria that lack complete programmatic ground truth. section 7.7.3

Financial research assistant

A verifier can check whether a response cites the source, matches the required JSON structure, and reproduces a revenue figure from the retrieved transcript. These checks convert otherwise manual scoring into explicit reward signals. section 7.7.3

Table 7.1 Task types mapped to reward types
Chapter 1 task type
Chapter 7 reward type
Verifiable task
Programmatic verifier
Open-ended task
Preference-trained reward model

Task type determines reward type: verifiable tasks use programmatic verifiers, while open-ended tasks use preference-trained reward models. section 7.1.1

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains task-selection criteria and concrete checks, but stops short of implementing a complete verifier.
Examples
Uses math answers, unit-tested code, regex-checked formats, and financial-assistant citation and schema checks.
What it assumes
Assumes the ability to distinguish objective, programmatically checkable tasks from subjective open-ended tasks.
What you can do after
Build from scratch objective checks that score mathematical, code, structural, or source-grounded outputs.
How it connects
Connects programmatic verification with RLVR, RLHF, reward models, and reference-guided grading.

Also explained in

Compare how 9 books explain verifier →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.