Reward model

As explained in Transformers in Action

A reward model is a learned model that assigns scores representing the desirability of outputs or actions according to preference feedback.

A reward model is a learned model that scores outputs or actions according to preference feedback. In RLHF, it supplies the preference signal used to fine-tune a language model’s policy toward human intent. section 5.2 section 5.1

The model can be trained for a specific task or adopted from a pretrained version. Reinforcement learning then adjusts the policy to maximize its scores, linking preference modeling to policy optimization. section 5.2

Reward models appear in several preference-alignment workflows:

• RLHF uses an explicit reward model as a foundational component for optimizing the language-model policy. section 5.2
• DPO removes the separate reward-model stage and directly optimizes the language model from preferred and less-preferred responses. section 5.2
• GRPO uses a reward model to score multiple completions, then compares scores within each prompt group instead of relying on a separate value function. section 5.2.5

Structured-completion rewards

A GRPO workflow can score completions according to whether they contain meaningful <think> and <answer> sections: complete structure receives full reward, partial structure receives partial reward, and invalid structure receives none. section 5.2.5

Figure 5.1 A simplified comparison of RLHF and DPO and how DPO directly optimizes for the policy without an explicit reward function

Compare RLHF’s explicit reward-model pathway with DPO’s direct policy optimization. section 5.2

How Transformers in Action covers it

How deep it goes

Explains the model’s role, alternatives, and grouped scoring workflow, but does not detail reward-model training internals.

Examples

Uses preferred responses, grouped completions, and a <think>/<answer> structure-checking reward function.

What it assumes

Assumes familiarity with language-model policies and reinforcement learning, while introducing preference labels and reward-guided alignment.

What you can do after

Apply reward-based alignment with RLHF or GRPO, or choose DPO when avoiding an explicit reward model.

How it connects

Connects reward models to RLHF, DPO, GRPO, PPO, preference labels, policy optimization, and value functions.

liveBookmanning.com

Read Transformers in Action on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Reinforcement Learning from Human FeedbackA reward model is a learned model that predicts the reward or preference value assigned to an output or behavior.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionA reward model is a model that assigns a scalar score representing the desirability of a response to an input.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA model that scores candidate outputs according to how well they match human preferences.
• 

  liveBookmanning.com
  Generative AI in ActionA reward model is a machine-learning model that assigns scores to outputs to estimate their quality or preference according to evaluation criteria.

Compare how 5 books explain reward model →

Related concepts

• reinforcement learning with human feedbackused by · Reinforcement learning with human feedback is a model-training approach that uses human evaluations or preferences to guide optimization of model behavior.
• Proximal Policy Optimizationused by · Proximal Policy Optimization is a policy-gradient reinforcement-learning algorithm that limits policy updates to keep successive policies close and promote stable learning.
• preference datauses · Preference data is a collection of judgments or comparisons indicating the relative preference or quality of alternatives, often model outputs.
• reinforcement learning with verifiable rewardscontrasts with · Reinforcement learning with verifiable rewards is a training approach that optimizes a model using rewards derived from objectively checkable outcomes.
• reinforcement learningused by · Reinforcement learning is a machine-learning paradigm in which an agent learns actions or a policy through interaction with an environment and evaluative feedback, such as rewards.

This explainer was generated by Manning's knowledge graph from the text of Transformers in Action. The section links point to the parts of the book it draws on.