A reward model (RM) is a learned proxy for human preferences. Given a prompt and a candidate completion, it produces a score that represents relative quality, turning judgments that are hard to specify programmatically into a signal that optimization can use. In RLHF, this learned signal replaces the fixed environment reward of standard reinforcement learning. section 5.0 section 3.1.3
The usual RLHF reward model is a Bradley–Terry preference model. For a prompt x and chosen and rejected completions y_c and y_r, it learns scores so that the probability of preferring the chosen answer is σ(rθ(y_c|x) − rθ(y_r|x)). Training minimizes −log σ(rθ(y_c|x) − rθ(y_r|x)): raise the chosen score, lower the rejected score, and care about their difference rather than any absolute score scale. section 5.1
From comparison to training signal
For a prompt with two generated answers, a labeler selects the more helpful answer. A language model with a scalar linear head scores both completions and is trained to rank the selected one higher. Later, the RM can score newly generated completions: select the top candidate in best-of-N sampling or return the terminal reward used to update a policy with RL. section 5.1 section 5.6.1
In the canonical pipeline, instruction fine-tuning first establishes a question-answering model. Preference comparisons across prompts, completions, and labelers then train an RM, often from that instruction-tuned checkpoint. RL samples fresh completions, obtains RM ratings, and updates the policy toward higher-scoring behavior. A frozen reference policy commonly supplies a KL-divergence penalty to limit drift during this optimization. section 1.3 section 6.0
Figure 6.1 Overview of the RLHF training loop. A prompt from the dataset is passed to the tuned policy, which generates a completion. The reward model scores this completion, and the frozen initial model (typically the instruction-tuned model before RL) computes log probabilities on the same text to calculate a Kullback–Leibler penalty that prevents excessive drift. The combined reward signal then drives an RL update to the policy parameters.
The loop shows where the RM sits operationally: it scores a policy completion, while a reference model supplies the KL penalty before the RL update. section 6.0
Reward-related models differ mainly by what they score and when that score is used:
- Bradley–Terry preference RM: scores a whole prompt-completion pair with one scalar; reranks candidates or supplies a terminal RL reward. section 5.6.1
- Outcome reward model: estimates per-token correctness in a completion, then aggregates token scores to assess a finished answer. section 5.6.1
- Process reward model: scores reasoning steps and can assess complete traces or prune low-scoring search branches. section 5.6.1
- Value function: predicts expected remaining return for a prefix and serves as an RL baseline, not as a correctness or preference score. section 5.6.1
A reward model gives flexibility because designers can learn the objective from preferences rather than accept a fixed reward function. That same flexibility creates a proxy-objective risk: RL can over-optimize the RM, including through undesirable tendencies such as excessive length or chattiness. Practical training therefore requires regularization and careful implementation; truncated generations can also produce out-of-distribution RM scores unless scoring and length penalties are handled deliberately. section 1.2 section 6.3
Reward modeling is not the only way to use preference data. Rejection sampling filters candidate completions with an RM, while direct alignment algorithms optimize a policy directly from pairwise preferences without first learning an intermediate RM. Where an RM is used for RL, policy-gradient methods translate its response-level scores into parameter updates over generated token sequences. section 3.1.5 section 6.6
