Chapter 5: Reward modeling
This introduction explains why reward models make human preferences usable for RLHF optimization and how a learned reward model supplies the agent’s learning signal.
After this chapter you will be able to
- Preferences as proxy objectives. Explain why reward models are central to RLHF: they turn hard-to-specify human preferences into a signal that downstream training can optimize.
- Learned reward signal. Describe how a reward model supplies a learning signal analogous to an RL environment, while being learned from human preferences rather than fixed. State that it produces a scalar reward and that the common Bradley–Terry model relates its prediction to preferred text in training comparisons.
- Bradley–Terry preference model. Explain how the Bradley–Terry model relates a pairwise preference to underlying item strengths or scores, and why only score differences affect the predicted preference.
- Pairwise reward training loss. Explain how the chosen and rejected completion scores form a preference probability and a negative log-likelihood loss, and why the log-sigmoid and softplus loss expressions are equivalent.
- Paired reward model inputs. Describe the chosen and rejected inputs a reward model needs, and explain why preparing those inputs is a substantial implementation task.
- Sequence-level scalar reward. Trace how the model produces one scalar reward for a prompt–completion sequence, including how it finds the last real token.
- Unsettled reward modeling practices. Explain that reward modeling is relatively under-explored in RLHF and that, although popular works have modified the traditional reward modeling loss, no modification has become a single best practice.
- Preference strength and margins. Explain what binarizing preference ratings discards and how a margin can represent the strength of a preference. Describe why using that extra information is not standard practice.
- Balanced comparisons per prompt. Explain why ranking multiple completions for one prompt requires balancing the resulting pairwise comparisons, and describe how grouping and averaging them addresses the problem.
- K-wise preference ranking. Describe how labelers rank K sampled actions for a prompt and explain how this K-wise setup relates to pairwise Bradley–Terry modeling.
- Per Token Correctness Training. Describe how the ORM head predicts correctness at each completion token and how training applies a binary cross-entropy loss using the completion’s outcome label while masking prompt tokens.
- ORM Versus Pairwise Models. Distinguish a per-token ORM trained on correctness labels from a Bradley–Terry model trained on correct-versus-incorrect response pairs, and note that some writing uses “ORM” more broadly.
- Step-Level Reward Supervision. Explain how a process reward model (PRM) scores a chain of thought and how its step-level target differs from a whole-response reward target.
- ORM and value function distinction. Distinguish what an outcome reward model (ORM) predicts from what a value function predicts, and explain how their training targets and uses differ.
- Whole-answer reward scores. Describe how a Bradley–Terry reward model scores a prompt and candidate completion, and how that score is used.
- Token and step correctness scores. Distinguish outcome and process reward models by what they score, how their scores can be combined, and what those scores are used for.
- Prefix value estimates. Explain what a value function predicts from a current prefix and how its values are used during RL training, in contrast to correctness scores.
- Language model judging setup. Explain why LLM-as-a-judge emerged and what information a language model receives when judging two completions.
- Generative judges and reward models. Describe how LLM-as-a-judge is used in post-training and distinguish its reported reward-model evaluation performance from that of existing reward models.
- Reward model evaluation landscape. Explain why RewardBench mattered to early reward-model research and distinguish the evaluation approaches the section names.
What it builds on
- Problem formulation (chapter 3) · Assumes the basic RL setup in which an agent receives a reward signal.
Section by section
Introduction
This introduction explains why reward models make human preferences usable for RLHF optimization and how a learned reward model supplies the agent’s learning signal.
- Preferences as proxy objectives: Explain why reward models are central to RLHF: they turn hard-to-specify human preferences into a signal that downstream training can optimize.
Why it matters: This explains why RLHF trains a reward model rather than requiring a complete, hand-written definition of good behavior. - Learned reward signal: Describe how a reward model supplies a learning signal analogous to an RL environment, while being learned from human preferences rather than fixed. State that it produces a scalar reward and that the common Bradley–Terry model relates its prediction to preferred text in training comparisons.
Why it matters: The distinction between a fixed environment reward and a learned preference signal clarifies what supplies feedback in RLHF.
Key terms: reward models, human preferences, proxy objectives, reward model, learning signal, scalar value of reward, Bradley–Terry reward model, pairwise preference relation
5.1 Training a Bradley–Terry reward model
This section explains how a Bradley–Terry reward model turns pairwise preferences into scalar scores and a training loss.
- Bradley–Terry preference model: Explain how the Bradley–Terry model relates a pairwise preference to underlying item strengths or scores, and why only score differences affect the predicted preference.
Why it matters: This explains what a reward model’s relative scores mean—and what they do not mean. - Scalar reward model scores: Describe how the standard reward model scores two completions for the same prompt, including the usual language-model modification that produces each score.
Why it matters: The training loss needs a comparable score for each completion. - Pairwise reward training loss: Explain how the chosen and rejected completion scores form a preference probability and a negative log-likelihood loss, and why the log-sigmoid and softplus loss expressions are equivalent.
Why it matters: These equivalent forms let you recognize the same reward-model objective in different implementations.
Key terms: Bradley–Terry (BT) model, latent strength, scores, sigmoid, reward model, scalar score, linear head, prompt–completion pair, chosen, rejected
5.2 Implementation example
This section shows how a reward model scores chosen and rejected prompt–completion pairs and turns a language model’s hidden state into a scalar reward.
- Paired reward model inputs: Describe the chosen and rejected inputs a reward model needs, and explain why preparing those inputs is a substantial implementation task.
Why it matters: Correctly prepared comparison pairs are necessary for the reward-modeling loss to use the intended inputs. - Sequence-level scalar reward: Trace how the model produces one scalar reward for a prompt–completion sequence, including how it finds the last real token.
Why it matters: This explains how a token-processing language model becomes a model that scores an entire response.
Key terms: inputs_chosen, inputs_rejected, data loader, inference pipeline, causal language model, attention_mask, output_hidden_states, last non-padding token, linear head, scalar reward
5.3 Reward model variants
This section introduces reward model variants as an under-explored area with no settled best practice for modifying the traditional loss.
- Unsettled reward modeling practices: Explain that reward modeling is relatively under-explored in RLHF and that, although popular works have modified the traditional reward modeling loss, no modification has become a single best practice.
Why it matters: It explains why the following variants are presented as alternatives rather than one standard recipe.
Key terms: reward modeling, traditional reward modeling loss, best practice
5.3.1 Preference margin loss
This subsection explains how preference-strength information can be retained in reward-model training instead of reducing ratings to a chosen–rejected pair.
- Preference strength and margins: Explain what binarizing preference ratings discards and how a margin can represent the strength of a preference. Describe why using that extra information is not standard practice.
Why it matters: It distinguishes knowing which answer won from knowing how decisively it won.
Key terms: Likert Scale, chosen and rejected completions, preference margin, m(y_c, y_r)
5.3.2 Balancing multiple comparisons per prompt
Explains why pairwise comparisons from the same prompt are grouped and averaged when training a reward model.
- Balanced comparisons per prompt: Explain why ranking multiple completions for one prompt requires balancing the resulting pairwise comparisons, and describe how grouping and averaging them addresses the problem.
Why it matters: Balancing the comparisons keeps prompts with many completions from dominating reward-model training.
Key terms: pairwise comparisons, reward model, training batch, grouped update
5.3.3 K-wise loss function
This subsection introduces a K-wise reward-model loss that uses rankings of multiple actions and relates it to pairwise Bradley–Terry modeling.
- K-wise preference ranking: Describe how labelers rank K sampled actions for a prompt and explain how this K-wise setup relates to pairwise Bradley–Terry modeling.
Why it matters: It shows how reward modeling can use a full ranking rather than only a pairwise preference.
Key terms: K-wise loss function, Plackett–Luce model, permutation, BT model
5.4 Outcome reward models
This section explains how outcome reward models learn to predict whether a completion’s final answer is correct, and distinguishes them from pairwise Bradley–Terry reward models.
- Per Token Correctness Training: Describe how the ORM head predicts correctness at each completion token and how training applies a binary cross-entropy loss using the completion’s outcome label while masking prompt tokens.
Why it matters: Per-token predictions are central to how this ORM differs from a sequence-level reward model. - ORM Versus Pairwise Models: Distinguish a per-token ORM trained on correctness labels from a Bradley–Terry model trained on correct-versus-incorrect response pairs, and note that some writing uses “ORM” more broadly.
Why it matters: The source of the labels alone does not tell you which kind of reward model was trained.
Key terms: ORM head, per-token, binary cross-entropy, mask, completion tokens, Bradley–Terry, BT model, chosen-rejected, contrastive, sequence-level loss, outcome reward model
5.5 Process reward models
This section explains how process reward models score individual reasoning steps and how step labels are used in training.
- Step-Level Reward Supervision: Explain how a process reward model (PRM) scores a chain of thought and how its step-level target differs from a whole-response reward target.
Why it matters: The distinction tells you what a PRM’s score is judging. - Binary Per-Step Training Objective: Describe the labels and prediction used in the section’s binary per-step loss.
Why it matters: It connects step annotations to the prediction the model is trained to make. - Boundary Labels and Masking: Describe where the three-class PRM places its labels and how non-boundary tokens are handled during loss computation.
Why it matters: The boundary placement explains how token-level model outputs become supervision for individual reasoning steps.
Key terms: process reward models, PRMs, reasoning step, process supervision, per-step cross-entropy loss, annotated steps, predicted probability, step boundaries, incorrect, neutral
5.6 Comparing reward model types (and value functions)
This section compares reward model types, focusing on how outcome reward models differ from value functions despite similar outputs, and where the boundaries between model types blur.
- ORM and value function distinction: Distinguish what an outcome reward model (ORM) predicts from what a value function predicts, and explain how their training targets and uses differ.
Why it matters: Similar-looking model outputs serve different purposes in training. - Reward model boundary cases: Explain why a discount factor of 1 does not make a value function an ORM, and describe when outcome-based supervision is still classified as a process reward model (PRM).
Why it matters: These cases show why output shape or the source of supervision alone does not settle a model's type.
Key terms: ORM, value function, offline labels, on-policy rollouts, advantages, discount factor, remaining-sum, PRM, per reasoning-step label
5.6.1 Inference across reward model types
This section compares what four model types take as input, what they output, and how their scores are used at inference time or during RL training.
- Whole-answer reward scores: Describe how a Bradley–Terry reward model scores a prompt and candidate completion, and how that score is used.
Why it matters: A single whole-answer score is used differently from the token- and step-level outputs of the other models. - Token and step correctness scores: Distinguish outcome and process reward models by what they score, how their scores can be combined, and what those scores are used for.
Why it matters: The unit being scored determines how to interpret and combine the scores. - Prefix value estimates: Explain what a value function predicts from a current prefix and how its values are used during RL training, in contrast to correctness scores.
Why it matters: A value estimate supports RL advantage calculations rather than directly judging whether an answer part is correct.
Key terms: Bradley–Terry RM, scalar, terminal reward, best-of-N sampling, Outcome RM, Process RM, per-token probabilities, step boundaries, aggregation, value function
5.7 Generative reward modeling (a.k.a. LLM-as-a-judge)
This section explains how language models can judge pairs of completions, why post-training recipes use them, and how they compare with reward models.
- Language model judging setup: Explain why LLM-as-a-judge emerged and what information a language model receives when judging two completions.
Why it matters: This is the basic setup behind using an LLM rather than a human labeler to judge completions. - Generative judges and reward models: Describe how LLM-as-a-judge is used in post-training and distinguish its reported reward-model evaluation performance from that of existing reward models.
Why it matters: Using LLM judges for preference data does not mean they have displaced reward models across evaluations. - Low temperature for consistent ratings: State the sampling-temperature tip for LLM-as-a-judge workflows and its intended effect.
Why it matters: More consistent ratings can make an LLM judging workflow more robust.
Key terms: LLM-as-a-judge, preference data, completions, generative reward models, RMs, post-training recipes, sampling temperature, variance of ratings
5.8 Further reading
This section points readers to reward-model benchmarks and training research while explaining how reward-model evaluation has broadened.
- Reward model evaluation landscape: Explain why RewardBench mattered to early reward-model research and distinguish the evaluation approaches the section names.
Why it matters: Different benchmarks can test different aspects of a reward model, so their results need context.
Key terms: RewardBench, RMs, LLM-as-a-judge
5.9 Summary
Where this is used later
- The role of reinforcement learning in RLHF (chapter 6) · Provides the reason a learned reward can guide RLHF optimization.
- AI feedback (chapter 12) · Introduces reward modeling as a stage of RLHF.
- Group relative policy optimization (chapter 6) · Introduces outcome reward models.
- Value functions and PPO (chapter 6) · The distinction supports the later discussion of value functions in policy optimization.
- Related: Best-of-N sampling (chapter 9) · Best-of-N sampling uses scores to choose among completed candidates.
- Building specific LLMs for judgment (chapter 12) · Provides the basic judging setup for discussing LLMs built specifically for judgment.
Concepts explained in this chapter
- outcome reward modelAn outcome reward model assigns a reward to the final outcome of a task or trajectory rather than to intermediate reasoning steps.
- reward modelA reward model is a learned model that predicts the reward or preference value assigned to an output or behavior.
- process reward modelA process reward model assigns rewards to intermediate steps or partial trajectories in a task or reasoning process.
- preference dataA pairwise comparison is an evaluation that determines which of two items is preferred.
- cross entropy lossCross-entropy is the expected negative log probability assigned by one distribution to outcomes drawn from another distribution.
- value functionA value network is a neural network that estimates the expected value of states or actions in reinforcement learning.
- Bradley-Terry modelA Bradley–Terry model is a probabilistic model that represents pairwise preferences using latent positive strengths or equivalent real-valued scores for items.
- negative log-likelihoodNegative log-likelihood is the negation of a model's log-likelihood, used as a loss for probabilistic predictions.
- evaluatorLLM-as-a-judge is an evaluation technique in which a language model assesses the quality or preference of model outputs according to provided criteria.
- generative reward modelA generative reward model is a reward model that generates evaluative judgments or preference assessments rather than only emitting a scalar score.
- preference marginA preference margin is the numerical difference in reward or rating values between two compared outputs, representing the strength of preference.
- preferenceA human preference is a person's comparative judgment about the desirability or quality of model behavior or outputs.
- overfittingOverfitting is a training condition in which a model fits its training data or objective too closely and generalizes poorly.
- output layerA scalar head is a model output component that maps hidden representations or logits to a single numerical prediction.
- reward signalA reward is a scalar evaluative signal indicating the desirability of an action or outcome for an agent.
- advantage functionAn advantage function estimates how much better an action is than the policy's expected value at a state.
- sigmoidThe sigmoid function maps a real-valued input to a number between zero and one using the function 1/(1+e^-x).
- alignment trainingValue alignment is the conformity of an intelligent system's objectives and behavior to the values held by the people or population it is intended to serve.
- inverse reinforcement learningInverse reinforcement learning is a technique that infers an agent's reward function from observed behavior or trajectories.
- proxy objectiveA proxy objective is an optimization objective that approximates a desired goal instead of measuring that goal directly.
- trajectoryAn episode is a finite sequence of interaction steps experienced by an agent during one trial.
- chain-of-thought promptingA reasoning trace is a sequence of generated textual intermediate steps used while solving a problem before reaching an answer.
- contrastive loss functionA contrastive loss is an optimization objective that compares examples to encourage similar or preferred examples to be distinguished from dissimilar or non-preferred ones.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.