Reward model
A reward model is a learned model that assigns scores to outputs, actions, or outcomes to represent their desirability according to preferences or evaluation criteria.
Reinforcement Learning from Human Feedback
A reward model is a learned model that predicts the reward or preference value assigned to an output or behavior.
- How it is used
- Supplies a learned preference signal for reranking completions and optimizing language-model policies in RLHF.
- How deep it goes
- Derives the Bradley–Terry pairwise loss and distinguishes sequence, token, step, and value-model outputs.
- Examples
- Uses chosen-versus-rejected completions, best-of-N reranking, response-level RL rewards, and trajectory preferences in Atari games.
- What you can do after
- Build it from scratch by attaching a scalar head and training it to rank chosen completions above rejected completions.
Read how Reinforcement Learning from Human Feedback explains reward model →
Deep Learning with R, Third Edition
A reward model is a model that assigns a scalar score representing the desirability of a response to an input.
- How it is used
- Uses ranked human preferences as a proxy for desirability when improving generated responses through reinforcement learning.
- How deep it goes
- Defines prompt–response scoring and outlines its training loop, but defers most reinforcement-learning details.
- Examples
- Uses ranked candidate responses, a smaller Transformer scorer, and reward-threshold pseudocode.
- What you can do after
- Apply it via reinforcement learning to score generated responses and select outputs receiving higher rewards.
Read how Deep Learning with R, Third Edition explains reward model →
Deep Learning with Python, Third Edition
A model that scores candidate outputs according to how well they match human preferences.
- How it is used
- Uses preference scores as a proxy for human feedback when improving a supervised-fine-tuned language model.
- How deep it goes
- Explains data collection, scoring architecture, and a basic reinforcement-learning loop, but defers deeper reinforcement-learning details.
- Examples
- Uses ranked prompt responses and pseudocode that filters generated responses by reward cutoff before fitting.
- What you can do after
- Apply it via reinforcement learning to score generated responses and guide further language-model tuning.
Read how Deep Learning with Python, Third Edition explains reward model →
Transformers in Action
A reward model is a learned model that assigns scores representing the desirability of outputs or actions according to preference feedback.
- How it is used
- Guides language-model policy optimization by scoring outputs against human preferences in reinforcement-learning alignment.
- How deep it goes
- Explains the model’s role, alternatives, and grouped scoring workflow, but does not detail reward-model training internals.
- Examples
- Uses preferred responses, grouped completions, and a <think>/<answer> structure-checking reward function.
- What you can do after
- Apply reward-based alignment with RLHF or GRPO, or choose DPO when avoiding an explicit reward model.
Generative AI in Action
A reward model is a machine-learning model that assigns scores to outputs to estimate their quality or preference according to evaluation criteria.
- How it is used
- Evaluates and ranks generated outputs so RLHF can optimize a language model toward human preferences for tasks such as chat, code, or creative writing.
- How deep it goes
- Explains response generation, human ranking, reward-model training, and downstream PPO use without deriving the model or algorithm mathematically.
- Examples
- Uses human rankings based on domain expertise, fact-checking, code execution, chat, code, and creative-writing quality.
- What you can do after
- Apply reward scores to understand how human preferences become a training signal for RLHF model improvement.
Related concepts
- reinforcement learning with human feedbackused by · Reinforcement learning with human feedback is a model-training approach that uses human evaluations or preferences to guide optimization of model behavior.
- Proximal Policy Optimizationused by · Proximal Policy Optimization is a policy-gradient reinforcement-learning algorithm that limits policy updates to keep successive policies close and promote stable learning.
- preference datauses · Preference data is a collection of judgments or comparisons indicating the relative preference or quality of alternatives, often model outputs.
- reinforcement learning with verifiable rewardscontrasts with · Reinforcement learning with verifiable rewards is a training approach that optimizes a model using rewards derived from objectively checkable outcomes.
- reinforcement learningused by · Reinforcement learning is a machine-learning paradigm in which an agent learns actions or a policy through interaction with an environment and evaluative feedback, such as rewards.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.