Reward model

As explained in Generative AI in Action

A reward model is a machine-learning model that assigns scores to outputs to estimate their quality or preference according to evaluation criteria.

A reward model is a separate model that scores and ranks outputs according to human judgments of quality and relevance. It converts preferences such as domain expertise, factual accuracy, or successful code execution into quantitative feedback for improving a language model. section 9.5.3 section 9.7

A reward model supports the RLHF workflow:

• Use an SFT model to generate multiple responses for each prompt. section 9.5.3
• Have human evaluators rank responses using criteria such as expertise, fact-checking, or code execution. section 9.5.3
• Train the reward model to predict those quality judgments and score new outputs. section 9.5.3 section 9.7
• Use its scores to further train the SFT model, often with PPO, producing an RLHF-aligned model. section 9.5.4 section 9.7

Support-response alignment

For a customer-support prompt, an SFT model generates several replies. Evaluators rank them for factual accuracy and relevance; the reward model learns these preferences and scores later replies so reinforcement learning can favor stronger responses. section 9.5.3 section 9.5.4

Figure 9.16 RLHF fine-tuning approaches to aligning language models [13]

Use the RLHF workflow diagram to trace how supervised fine-tuning, reward modeling, PPO, and human feedback work together. section 9.7

How Generative AI in Action covers it

How deep it goes

Explains response generation, human ranking, reward-model training, and downstream PPO use without deriving the model or algorithm mathematically.

Examples

Uses human rankings based on domain expertise, fact-checking, code execution, chat, code, and creative-writing quality.

What it assumes

Assumes familiarity with an SFT model and human preference rankings while situating reward modeling within RLHF.

What you can do after

Apply reward scores to understand how human preferences become a training signal for RLHF model improvement.

How it connects

Connects supervised fine-tuning, reinforcement learning, RLHF, PPO, human feedback, and alignment.

liveBookmanning.com

Read Generative AI in Action on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Reinforcement Learning from Human FeedbackA reward model is a learned model that predicts the reward or preference value assigned to an output or behavior.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionA reward model is a model that assigns a scalar score representing the desirability of a response to an input.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA model that scores candidate outputs according to how well they match human preferences.
• 

  liveBookmanning.com
  Transformers in ActionA reward model is a learned model that assigns scores representing the desirability of outputs or actions according to preference feedback.

Compare how 5 books explain reward model →

Related concepts

• reinforcement learning with human feedbackused by · Reinforcement learning with human feedback is a model-training approach that uses human evaluations or preferences to guide optimization of model behavior.
• Proximal Policy Optimizationused by · Proximal Policy Optimization is a policy-gradient reinforcement-learning algorithm that limits policy updates to keep successive policies close and promote stable learning.
• preference datauses · Preference data is a collection of judgments or comparisons indicating the relative preference or quality of alternatives, often model outputs.
• reinforcement learning with verifiable rewardscontrasts with · Reinforcement learning with verifiable rewards is a training approach that optimizes a model using rewards derived from objectively checkable outcomes.
• reinforcement learningused by · Reinforcement learning is a machine-learning paradigm in which an agent learns actions or a policy through interaction with an environment and evaluative feedback, such as rewards.

This explainer was generated by Manning's knowledge graph from the text of Generative AI in Action. The section links point to the parts of the book it draws on.