Reward model training turns human preference comparisons into scores that guide later reinforcement learning. Given a prompt and response, a transformer-based model produces one scalar score estimating how good the complete response is. section 10.6 section 10.4
Train the model to rank responses, not to hit fixed score targets:
- Pass both the preferred and rejected responses through the model. section 10.6
- Use a ranking loss that grows when the preferred response scores too close to or below the rejected one; updates push the preferred score up and the rejected score down. section 10.6
- Encourage a minimum score gap between responses. The particular score values matter less than their ordering. section 10.6
Preferences depend on the task
For a clinical decision-support tool, a response listing several plausible causes may be preferred. Different preference data can therefore teach different response behavior. section 10.6
Figure 10.5 Learning a reward model from human preferences.

Visualizes preferred and rejected responses receiving scores, where their ordering matters more than the exact values. section 10.6
