A reward model learns to predict human preferences from comparisons between responses. In RLHF, PPO uses its predictions as a training signal to optimize a policy while keeping it close to a reference policy. section 8.4.1 section 8.4
The reward-model stage brings practical constraints:
- Collecting robust preference data can require tens of thousands of annotated pairs; disagreement between annotators makes multiple labels per pair useful. section 8.4.1
- The model must generalize to unseen prompts. If it overfits, the policy can exploit its errors instead of improving responses. section 8.4.1
Figure 8.6 RLHF’s three stages and four models versus DPO’s single supervised step.

Compare the reward-model stage in RLHF with DPO’s single supervised step. section 8.4.1









