Chapter 10: The nature of preferences
The introduction explains why RLHF centers human preferences when a reward function is difficult to specify.
After this chapter you will be able to
- Human preferences in RLHF. Explain why RLHF emerged and how human preferences figure in it: specifying a reward function can be hard, so preferences become both what the system tries to model and a source of training data.
- Preference versus factual correctness. Distinguish a preference judgment between two poems from a question with a clear right answer, and explain why human feedback is used as a reward signal in RLHF.
- Foundations of preference optimization. Explain that using RL to optimize a reward model of preferences brings together ideas from several fields and rests on assumptions about human preferences.
- Preferences as optimization objectives. Explain why RLHF must connect human preferences to rewards: its optimization uses reward models, while preferences, rewards, and costs are different ways to express relative goodness.
- Single rewards versus human desires. Explain the tension between RL’s use of rewards to direct behavior toward clearly defined goals and reward models that aggregate many human preferences into one function.
- Bellman and MDP foundations. Explain how modern reinforcement learning draws on the Bellman equation and the Markov decision process formulation to estimate future reward, and why that formulation matters for performance guarantees.
- Defined rewards versus preferences. Distinguish the closed or explicitly rewarded tasks in which RL methods have shown success from the problem of optimizing models of human preferences.
- Stable rewards and shifting preferences. Explain why a reward model’s scalar output cannot simply be treated as a single, stable, true human preference.
- Preference measurement and context. Explain how the way choices are presented and measured can change the preferences recorded in RLHF data.
What it builds on
- What does RLHF do? (chapter 1) · Introduces the purpose of RLHF.
- Training a Bradley–Terry reward model (chapter 5) · The use of a reward model to represent preferences.
- The role of reinforcement learning in RLHF (chapter 6) · The role of RL in optimization.
Section by section
Introduction
The introduction explains why RLHF centers human preferences when a reward function is difficult to specify.
- Human preferences in RLHF: Explain why RLHF emerged and how human preferences figure in it: specifying a reward function can be hard, so preferences become both what the system tries to model and a source of training data.
Why it matters: This explains why preference data is central to RLHF rather than an optional addition.
Key terms: RLHF, human preferences, reward function
10.1 When preference replaces correctness
This section contrasts judging creative outputs by preference with checking factual correctness, and explains why RLHF uses human feedback as an indirect reward signal.
- Preference versus factual correctness: Distinguish a preference judgment between two poems from a question with a clear right answer, and explain why human feedback is used as a reward signal in RLHF.
Why it matters: It explains why RLHF needs preference judgments when correctness alone cannot settle which output is better. - Cross-disciplinary preference assumptions: Describe the three broad areas that converge in modern RLHF and explain what their differing assumptions contribute to RLHF problem design.
Why it matters: These different starting assumptions help explain why defining and optimizing preferences is not straightforward. - Empirical alignment and values: Distinguish the section’s practical emphasis on empirical alignment from measuring alignment to specific values, while noting the continuing interest in pluralistic alignment.
Why it matters: It keeps practical performance measures distinct from the broader question of whose values a model reflects.
Key terms: human preferences, reward signals, target reward, maximizing utility, modern deep learning systems, empirical alignment, specific skills, pluralistic alignment
10.2 The origins of RLHF and preferences
This section frames modern RLHF as a meeting of ideas about human preferences, reward models, and reinforcement learning, while noting a shift from exact solutions toward approximate ones.
- Foundations of preference optimization: Explain that using RL to optimize a reward model of preferences brings together ideas from several fields and rests on assumptions about human preferences.
Why it matters: It makes clear that preference optimization depends on assumptions about what is being measured, not just on an RL algorithm. - Shift toward approximate solutions: Describe the section’s claim that modern problem specifications have moved away from settings with optimal solutions and use underspecified models as approximations.
Why it matters: It explains why uncertainty in the objective is part of the setting this chapter examines.
Key terms: RL, reward model (RM), preferences, human values, optimal solutions, underspecified models, approximate solutions
10.3 Specifying objectives: From logic of utility to reward functions
The section connects preferences, rewards, and costs as ways to specify an optimization objective, then contrasts RL’s assumptions about rewards with the difficulty of representing human preferences in one reward model.
- Preferences as optimization objectives: Explain why RLHF must connect human preferences to rewards: its optimization uses reward models, while preferences, rewards, and costs are different ways to express relative goodness.
Why it matters: This connection explains why questions about the meaning of preferences matter to RLHF optimization. - Limits of preference measurement: Describe why measuring human preferences is contested, including the view that a preference model’s accuracy depends on its task and social context, and distinguish those objections from assumptions used in RLHF.
Why it matters: A reward model’s target should not be mistaken for an uncontested measurement of what people value. - Reward-to-go and future rewards: Explain reward-to-go as cumulative future reward in a trial and describe how RL uses a discount factor to reweight future rewards when optimizing toward defined goals.
Why it matters: The conventional RL objective helps reveal what is being assumed when human judgments become rewards. - Single rewards versus human desires: Explain the tension between RL’s use of rewards to direct behavior toward clearly defined goals and reward models that aggregate many human preferences into one function.
Why it matters: It identifies a central difficulty in using standard RL optimization for human preferences.
Key terms: RMs, preferences, rewards, cost landscapes, preference measurement, social relations, preference model, reward-to-go, discount factor, utility-to-go
10.4 Tools for optimizing utility
This section traces how reinforcement learning tools developed for optimizing defined rewards and explains why applying them to human preferences is less straightforward.
- Bellman and MDP foundations: Explain how modern reinforcement learning draws on the Bellman equation and the Markov decision process formulation to estimate future reward, and why that formulation matters for performance guarantees.
Why it matters: It identifies the assumptions behind the performance claims associated with these tools. - Trial-and-error value learning: Describe how trial-and-error learning and temporal-difference updates shaped modern RL, and distinguish the future-reward estimates given by a Q-function and a value function.
Why it matters: It connects the section's history of RL to the estimates and updates its later methods use. - Defined rewards versus preferences: Distinguish the closed or explicitly rewarded tasks in which RL methods have shown success from the problem of optimizing models of human preferences.
Why it matters: Success on a task with a defined optimum does not by itself establish the same result for human preferences. - Inverse reinforcement learning connection: State what inverse reinforcement learning tries to learn and explain its connection to reward modeling in RLHF.
Why it matters: It points to a related way of thinking about learned rewards without treating it as part of the established RLHF recipe.
Key terms: Bellman equation, reward-to-go, Markov decision process (MDP), trial-and-error learning, temporal-difference (TD) learning, Q-learning, Q-function, value function, utility, closed-form reward function
10.5 Complexity of optimizing preferences
This section explains why optimizing human preferences is harder than optimizing a fixed reward function, while emphasizing that these difficulties do not make RLHF useless.
- Stable rewards and shifting preferences: Explain why a reward model’s scalar output cannot simply be treated as a single, stable, true human preference.
Why it matters: It explains why a reward score is a useful approximation, not a definitive account of what people want. - Formal utility and its limits: Describe what VNM utility theory offers for representing preferences and why that theoretical permission does not settle preference modeling in RLHF.
Why it matters: A mathematical basis for utility functions does not guarantee that one captures the preferences encountered in RLHF. - Preference measurement and context: Explain how the way choices are presented and measured can change the preferences recorded in RLHF data.
Why it matters: How feedback is collected can affect what a preference model is asked to learn. - Limits of preference aggregation: Explain why combining preferences across people, times, and situations creates tradeoffs rather than one universally fair optimization target.
Why it matters: A dataset that combines many judgments must contend with differences it cannot simply average away. - Practical limits of RLHF: Distinguish the claim that preference optimization cannot be fully solved from the claim that RLHF cannot work in practice.
Why it matters: It keeps limitations of preference models in perspective without dismissing their practical use.
Key terms: reward model, scalar signal, preferences, value, Von Neumann–Morgenstern (VNM) utility theorem, expected value, Markovian formulation, measured preferences, proxy measurements, context
10.6 Summary
Where this is used later
- Why we need preference data (chapter 11) · Motivates the need for preference data.
- Collecting preference data (chapter 11) · The distinction between preference and correctness.
- Open questions in RLHF preference data (chapter 11) · Provides context for open questions about preference data.
- (chapter intro) (chapter 11) · The discussion of specifying objectives provides context for the difficulty of expressing preferences as a reward function.
- Managing proxy objectives (chapter 14) · Provides context for the limitations of a proxy objective.
- Product cycles and what’s next for RLHF (chapter 17) · The complexity of optimizing preferences.
- Bias: Things to watch out for in data collection (chapter 11) · Motivates attention to biases in collected feedback.
Concepts explained in this chapter
- preferenceA human preference is a person's comparative judgment about the desirability or quality of model behavior or outputs.
- reward modelA reward model is a learned model that predicts the reward or preference value assigned to an output or behavior.
- reinforcement learning with human feedbackReinforcement learning from human feedback (RLHF) is a model-alignment technique that optimizes behavior using human preference signals, often through a learned reward model or direct preference objective.
- reward signalA reward is a scalar evaluative signal indicating the desirability of an action or outcome for an agent.
- reinforcement learningReinforcement learning is a machine learning paradigm in which an agent learns behavior by optimizing rewards obtained through interaction with an environment.
- reward-to-goReward-to-go is the cumulative future reward expected from a specified point in a trial or trajectory.
- Von Neumann-Morgenstern utility theoremThe Von Neumann–Morgenstern utility theorem states conditions under which rational preferences over lotteries can be represented by an expected-utility function.
- preference aggregationPreference aggregation is the process of combining preference judgments from multiple people, contexts, or situations into a collective representation.
- Bellman equationA Bellman equation recursively relates a value or action-value function to immediate reward and the value of successor states.
- Markov decision processA Markov decision process is a mathematical framework that models states, actions, rewards, and probabilistic transitions for sequential decision making under the Markov property.
- action-value functionA Q-function maps a state-action pair to a scalar estimate of expected future reward.
- inverse reinforcement learningInverse reinforcement learning is a technique that infers an agent's reward function from observed behavior or trajectories.
- accuracyTraining accuracy is the proportion of training examples for which a model produces the correct output.
- proxy metricA proxy metric is an imperfect measurable quantity used to approximate a desired quality or objective.
- value functionA value network is a neural network that estimates the expected value of states or actions in reinforcement learning.
- impossibility theoremAn impossibility theorem proves that no rule can simultaneously satisfy a specified set of desirable conditions.
- discount factorA discount factor is a scalar between zero and one that weights future rewards relative to nearer rewards.
- preference dataA pairwise comparison is an evaluation that determines which of two items is preferred.
- objective functionA loss function is a scalar objective that quantifies prediction or optimization error during model training.
- policyA policy is a function that maps an environment state to a probability distribution over actions.
- alignment trainingValue alignment is the conformity of an intelligent system's objectives and behavior to the values held by the people or population it is intended to serve.
- styleResponse style is the characteristic manner in which a model presents content, including presentation and rhetorical qualities such as bluntness or warmth.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.