Reward model
A reward model is a learned model that assigns scores to outputs, actions, or outcomes to represent their desirability according to preferences or evaluation criteria.
Reinforcement Learning from Human Feedback
A reward model is a learned model that predicts the reward or preference value assigned to an output or behavior.
- How it is used
- Uses learned preference scores as a proxy reward for language-model reinforcement learning and candidate selection.
- How deep it goes
- Derives the Bradley–Terry pairwise loss, describes scalar-head implementation and inference, and compares reward-model variants.
- Examples
- Uses the InstructGPT pipeline and chosen-versus-rejected completions to illustrate preference-based scoring and optimization.
- What you can do after
- Applies AutoModelForSequenceClassification to construct a scalar preference scorer for reranking or RLHF.
Read how Reinforcement Learning from Human Feedback explains reward model →
LLM Evaluation and Alignment, The Foundational Ideas
A reward model is a learned model that predicts the desirability or human preference of an AI system's output.
- How it is used
- Provides a learned reward signal for aligning policies on open-ended tasks where human judgment substitutes for checkable ground truth.
- How deep it goes
- Explains scalar-head architecture, pairwise ranking, policy optimization, failure modes, and the contrast between outcome and process supervision.
- Examples
- Uses ranked earnings summaries and step-labeled math solutions to illustrate response-level and process-level rewards.
- What you can do after
- Evaluate reward signals by task verifiability, labeling quality, reward-hacking risks, and the need for step-level feedback.
Read how LLM Evaluation and Alignment, The Foundational Ideas explains reward model →
Applied Reinforcement Learning
A reward model predicts a numerical score for an output to represent its desirability or alignment with preferences.
- How it is used
- Uses preference scores as scalable feedback for aligning a medical support assistant through reinforcement learning.
- How deep it goes
- Explains pairwise ranking, a transformer reward head, validation, and PPO integration with core implementation listings, omitting trainer boilerplate.
- Examples
- Uses chest-pain response comparisons and a headache-query score-gap calculation to illustrate preferences and ranking loss.
- What you can do after
- Apply reward modeling with PyTorch and Hugging Face Transformers to score preference pairs and support PPO fine-tuning.
Read how Applied Reinforcement Learning explains reward model →
Build a DeepSeek Model (From Scratch)
A reward model is a learned model that scores outputs according to preference-based or otherwise non-mechanical judgments of desired behavior.
- How it is used
- Guides post-training rewards for open-ended behavior when deterministic verification cannot capture helpfulness, harmlessness, nuance, or style.
- How deep it goes
- Distinguishes learned preference signals from deterministic verifiers and places them in hybrid reinforcement learning without detailing model internals.
- Examples
- Uses helpfulness and harmlessness prompts alongside math and code tasks where rule-based checkers provide the alternative.
- What you can do after
- Apply hybrid reward design by using learned preference signals for open-ended tasks and exact checkers for verifiable tasks.
Read how Build a DeepSeek Model (From Scratch) explains reward model →
Understanding AI
A reward model is a learned model that scores outputs according to preferences inferred from human or other evaluative feedback.
- How it is used
- Shapes LLM behavior during alignment training by converting human rankings into a learned preference signal.
- How deep it goes
- Defines the scoring role and traces rankings through reward-model learning to LLM updates, but stops short of implementation details.
- Examples
- Contrasts concise explanations, confusing answers, polite refusals, and dangerous instructions through reviewer rankings.
- What you can do after
- Evaluate how human rankings become learned preferences that guide an LLM toward more helpful, safe, and trustworthy responses.
Deep Learning with R, Third Edition
A reward model is a model that assigns a scalar score representing the desirability of a response to an input.
- How it is used
- Uses ranked human preferences as a proxy for desirability when improving generated responses through reinforcement learning.
- How deep it goes
- Defines prompt–response scoring and outlines its training loop, but defers most reinforcement-learning details.
- Examples
- Uses ranked candidate responses, a smaller Transformer scorer, and reward-threshold pseudocode.
- What you can do after
- Apply it via reinforcement learning to score generated responses and select outputs receiving higher rewards.
Read how Deep Learning with R, Third Edition explains reward model →
Deep Learning with Python, Third Edition
A model that scores candidate outputs according to how well they match human preferences.
- How it is used
- Uses preference scores as a proxy for human feedback when improving a supervised-fine-tuned language model.
- How deep it goes
- Explains data collection, scoring architecture, and a basic reinforcement-learning loop, but defers deeper reinforcement-learning details.
- Examples
- Uses ranked prompt responses and pseudocode that filters generated responses by reward cutoff before fitting.
- What you can do after
- Apply it via reinforcement learning to score generated responses and guide further language-model tuning.
Read how Deep Learning with Python, Third Edition explains reward model →
Transformers in Action
A reward model is a learned model that assigns scores representing the desirability of outputs or actions according to preference feedback.
- How it is used
- Guides language-model policy optimization by scoring outputs against human preferences in reinforcement-learning alignment.
- How deep it goes
- Explains the model’s role, alternatives, and grouped scoring workflow, but does not detail reward-model training internals.
- Examples
- Uses preferred responses, grouped completions, and a <think>/<answer> structure-checking reward function.
- What you can do after
- Apply reward-based alignment with RLHF or GRPO, or choose DPO when avoiding an explicit reward model.
Generative AI in Action
A reward model is a machine-learning model that assigns scores to outputs to estimate their quality or preference according to evaluation criteria.
- How it is used
- Evaluates and ranks generated outputs so RLHF can optimize a language model toward human preferences for tasks such as chat, code, or creative writing.
- How deep it goes
- Explains response generation, human ranking, reward-model training, and downstream PPO use without deriving the model or algorithm mathematically.
- Examples
- Uses human rankings based on domain expertise, fact-checking, code execution, chat, code, and creative-writing quality.
- What you can do after
- Apply reward scores to understand how human preferences become a training signal for RLHF model improvement.
LLM Customization and Fine-Tuning
A model trained to assign scores representing the desirability of outputs, often from human preference judgments.
- How it is used
- Supplies PPO with a learned preference signal for optimizing a policy while it stays near a reference policy.
- How deep it goes
- Explains preference-pair training, generalization risk, and compute needs, but does not specify model architecture or reward objective.
- Examples
- Uses paired human comparisons and contrasts RLHF’s reward-model stage with DPO’s single supervised step.
- What you can do after
- Understand how preference labels become a PPO training signal and how reward-model overfitting can misdirect policy optimization.
Read how LLM Customization and Fine-Tuning explains reward model →
Also covered in
- Fight Fraud with Machine LearningIn early access (MEAP)
- Build a Reasoning Model (From Scratch)
- How Large Language Models Work
- Build AI Drug Discovery PipelinesIn early access (MEAP)
- Introduction to Generative AI, Second Edition
- Grokking AI Algorithms, Second Edition
- Machines that ThinkIn early access (MEAP)
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- fine-tuningFine-tuning is the continued training of a pretrained model on targeted data or objectives to adapt its parameters or behavior to a task, domain, or use.
- reinforcement learningReinforcement learning is a machine-learning paradigm in which an agent learns behavior or a decision policy through interaction with an environment and evaluative feedback, such as rewards.
- RLHFReinforcement learning from human feedback is a model-training approach that uses human evaluations or preferences as feedback to optimize model behavior.
Related concepts
- RLHFused by · Reinforcement learning from human feedback is a model-training approach that uses human evaluations or preferences as feedback to optimize model behavior.
- preference datauses · Preference data is a collection of judgments about the relative preference or quality of alternatives, often candidate outputs for the same input.
- Proximal Policy Optimizationused by · Proximal Policy Optimization is a policy-gradient reinforcement-learning algorithm that limits policy updates to keep successive policies close and promote stable learning.
- reinforcement learning with verifiable rewardscontrasts with · Reinforcement learning with verifiable rewards is a training approach that uses rewards derived from objectively checkable outcomes to optimize a model or agent.
- reward signalproduces · A reward signal is a value or feedback indicating the desirability, benefit, or cost associated with an action, outcome, or behavior.
- reinforcement learningused by · Reinforcement learning is a machine-learning paradigm in which an agent learns behavior or a decision policy through interaction with an environment and evaluative feedback, such as rewards.
Questions about reward model
What is reward model?
A reward model is a learned model that assigns scores to outputs, actions, or outcomes to represent their desirability according to preferences or evaluation criteria.
Which Manning books explain reward model?
Reinforcement Learning from Human Feedback, LLM Evaluation and Alignment, The Foundational Ideas, Applied Reinforcement Learning, Build a DeepSeek Model (From Scratch), Understanding AI, Deep Learning with R, Third Edition, Deep Learning with Python, Third Edition, Transformers in Action and 2 more.
What should I know before learning reward model?
Manning books that teach reward model usually introduce language model, fine-tuning, reinforcement learning and RLHF first.
What is the difference between reward model and reinforcement learning with verifiable rewards?
A reward model is a learned model that assigns scores to outputs, actions, or outcomes to represent their desirability according to preferences or evaluation criteria. Reinforcement learning with verifiable rewards is a training approach that uses rewards derived from objectively checkable outcomes to optimize a model or agent.
What is reward model also called?
Reward model is also called preference model.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.









