liveBook · Manning

Reward model

As explained in LLM Customization and Fine-Tuning

A model trained to assign scores representing the desirability of outputs, often from human preference judgments.

A reward model learns to predict human preferences from comparisons between responses. In RLHF, PPO uses its predictions as a training signal to optimize a policy while keeping it close to a reference policy. section 8.4.1 section 8.4

The reward-model stage brings practical constraints:

  • Collecting robust preference data can require tens of thousands of annotated pairs; disagreement between annotators makes multiple labels per pair useful. section 8.4.1
  • The model must generalize to unseen prompts. If it overfits, the policy can exploit its errors instead of improving responses. section 8.4.1
Figure 8.6 RLHF’s three stages and four models versus DPO’s single supervised step.

Compare the reward-model stage in RLHF with DPO’s single supervised step. section 8.4.1

How LLM Customization and Fine-Tuning covers it

How deep it goes
Explains preference-pair training, generalization risk, and compute needs, but does not specify model architecture or reward objective.
Examples
Uses paired human comparisons and contrasts RLHF’s reward-model stage with DPO’s single supervised step.
What you can do after
Understand how preference labels become a PPO training signal and how reward-model overfitting can misdirect policy optimization.
How it connects
Connects human preference data, PPO, policy models, reference policies, and direct preference optimization (DPO).

Also explained in

Compare how 10 books explain reward model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.