liveBook · Manning

Reward model training

As explained in Applied Reinforcement Learning

Reward modeling trains a model to estimate the desirability of responses from preference comparisons.

Reward model training turns human preference comparisons into scores that guide later reinforcement learning. Given a prompt and response, a transformer-based model produces one scalar score estimating how good the complete response is. section 10.6 section 10.4

Train the model to rank responses, not to hit fixed score targets:

  • Pass both the preferred and rejected responses through the model. section 10.6
  • Use a ranking loss that grows when the preferred response scores too close to or below the rejected one; updates push the preferred score up and the rejected score down. section 10.6
  • Encourage a minimum score gap between responses. The particular score values matter less than their ordering. section 10.6

Preferences depend on the task

For a clinical decision-support tool, a response listing several plausible causes may be preferred. Different preference data can therefore teach different response behavior. section 10.6

Figure 10.5 Learning a reward model from human preferences.

Visualizes preferred and rejected responses receiving scores, where their ordering matters more than the exact values. section 10.6

How Applied Reinforcement Learning covers it

How deep it goes
Explains paired-response ranking, score margins, and gradient updates without requiring exact target scores.
Examples
Uses preferred-versus-rejected response pairs and a clinical example favoring multiple plausible causes.
What you can do after
Understand how to train a model to rank preferred responses above rejected ones.
How it connects
Connects human preference data with supervised fine-tuning and reinforcement learning.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.