liveBook · Manning

Reward model

As explained in LLM Evaluation and Alignment, The Foundational Ideas

A reward model is a learned model that predicts the desirability or human preference of an AI system's output.

When an output has no single checkable answer, how can a training system judge it? A reward model learns to predict desirability from preference labels. For a language model, it can take a prompt and response and return a scalar score that approximates what labelers prefer—not a guarantee of actual quality. section 7.1.1 section 7.5.1

A typical preference-based training pipeline has three stages:

  • Fine-tune a pretrained model on demonstrations of desired responses. section 7.5.1
  • Sample several responses per prompt, have labelers rank them, and train a reward model on the resulting comparisons. A common architecture replaces a transformer's language-model head with a scalar output head. section 7.5.1 section 7.5.3
  • Sample new responses and use their reward scores to update the policy with reinforcement learning. A KL penalty can keep the policy close to the fine-tuned reference model, where reward predictions are more reliable. section 7.5.1

For each preferred and less-preferred response, a Bradley–Terry ranking objective trains the model to give the preferred response a higher score. Many comparisons teach an aggregate preference signal; they do not explicitly identify which words or reasoning steps caused each judgment. section 7.4.2 section 7.4.3

Ranking earnings summaries

For a financial research prompt, a labeler ranks five summaries: a sourced summary leading with revenue first, a grounded summary burying the headline second, and summaries with unsupported claims, generic filler, or invented details lower. The five-way ranking yields ten response pairs for training. Across many rankings, the reward model can learn preferences such as citing sources and avoiding invented claims. section 7.5.3

Figure 7.8 The InstructGPT three-stage pipeline. Stage 1 fine-tunes the pretrained model on labeler-written demonstrations. Stage 2 trains a reward model on labeler-ranked comparisons of model outputs. Stage 3 optimizes the SFT model with reinforcement learning using PPO, with the reward model's score (penalized by KL divergence from the SFT model) as the reward signal. The SFT model from stage 1 plays two roles: it initializes the RL policy and is the reference distribution in the KL penalty. Source: Figure 2 of Ouyang et al. 2022.

Follow the flow from demonstrations to preference scoring and policy optimization, including the reference model used for KL control. section 7.5.1

A high reward score can be misleading. A policy may exploit shortcuts in the learned approximation, producing verbose, agreeable, or superficially well-formatted responses without improving their substance. Length normalization or a length penalty can address verbosity; adversarial preference data can teach the model to favor corrections over agreement. Labeler selection and clear rubrics also matter: a narrow or inconsistent labeler pool can embed its biases in the score. section 7.6.1 section 7.6.2

Choose the reward signal for the task. When an answer can be checked against a known value or by running code, a programmatic verifier may be preferable to a learned preference model. For multi-step reasoning, an outcome reward model scores the final response, while a process reward model predicts labels for intermediate steps. Step-level feedback helps locate errors but costs more to label. section 7.1.1 section 7.7.1

Human rankings are not the only possible labels: Constitutional AI uses an AI judge guided by a principle to choose between responses, then trains a reward model on those preferences. Direct Preference Optimization is another alternative that trains directly on preference pairs without an explicit reward model. section 8.2.4 section 7.5.6

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains scalar-head architecture, pairwise ranking, policy optimization, failure modes, and the contrast between outcome and process supervision.
Examples
Uses ranked earnings summaries and step-labeled math solutions to illustrate response-level and process-level rewards.
What you can do after
Evaluate reward signals by task verifiability, labeling quality, reward-hacking risks, and the need for step-level feedback.
How it connects
Connects preference learning to reinforcement learning from human feedback, reward hacking, KL divergence, process supervision, and verifiable rewards.

Also explained in

Compare how 10 books explain reward model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.