Chapter 11: Preference data
The introduction explains why preference data is central to RLHF and identifies two challenges in collecting it.
After this chapter you will be able to
- Preference data as proxy. Explain why RLHF uses preference data instead of a precisely specified reward function, and what that data helps models do.
- Preference data for reward models. Explain why RLHF collects preference data to train reward models rather than trying to capture complex human values directly in a single reward function.
- Comparisons over answer generation. Explain why distinguishing a good answer from a bad one makes preference data practical to collect.
- On-policy preference data. Describe what on-policy means for preference data and why collecting it from related model checkpoints helps RLHF training.
- Relative meaning of chosen. Distinguish a chosen answer in preference data from an answer that is globally correct.
- Interface effects on judgments. Explain why the collection interface matters and why its effects are hard to predict, using response speed as the section’s example.
- Paired-answer collection settings. Describe how an interface collects a preference between two answers, and distinguish dedicated labeling from comparisons shown to everyday users.
- Rankings and ratings distinction. Distinguish rankings, which order model completions relative to one another, from ratings, which score each piece of text. State that training commonly uses rankings, while ratings may serve as metadata.
- Likert-scale relative preferences. Explain how a Likert scale records which response is preferred, and why its integer value does not make it an individual rating. Describe the section’s even-scale example and its eventual reduction to a binary reward-model signal.
- Multiturn Preference Collection. Describe where preferences may be collected in a multiturn conversation and how the conversation proceeds when every response is judged.
- Conversation Turns as Training Prompts. Explain how turns from a multiturn conversation can be used in training and what cautions the section gives.
- Automatically checkable preferences. Explain how a domain with checkable outcomes can supply chosen and rejected responses without a subjective quality judgment.
- Constraint-based comparison construction. Describe how the poem example constructs a preference pair and why the comparison prompt must retain the constraint.
- Vendor access and contracts. Explain why obtaining preference data from a vendor can be slow or uncertain, and identify contractual terms a buyer needs to check.
- Iterative batch collection. Trace how weekly deliveries, early calibration, and training feedback are meant to improve preference data during a contract.
- Bias in preference data. Explain how bias in preference data can affect the final model, using prefix bias as an example.
- Sparse comparisons and broad preferences. Explain why preference-data curation and its impacts deserve scrutiny: individual comparisons between two texts provide limited evidence for the broad, complex preferences RLHF aims to capture.
- Collection and representation questions. Describe the open questions about whether collection setting, feedback format, and annotator population shape the preferences recorded and their effects on models.
- Annotator intent and model behavior. Explain the unresolved question of whether trained model behavior matches the preferences or specifications given to annotators, and why the section says this is hard to check.
What it builds on
- Specifying objectives: From logic of utility to reward functions (chapter 10) · The discussion of specifying objectives provides context for the difficulty of expressing preferences as a reward function.
- Fine-tuning and regularization (chapter 3) · The idea that post-training proceeds through model fine-tuning stages.
- When preference replaces correctness (chapter 10) · The distinction between preference and correctness.
- Implementation details (chapter 4) · The instruction-fine-tuning treatment of masking earlier conversation turns from the loss.
Section by section
Introduction
The introduction explains why preference data is central to RLHF and identifies two challenges in collecting it.
- Preference data as proxy: Explain why RLHF uses preference data instead of a precisely specified reward function, and what that data helps models do.
Why it matters: This explains why collecting preferences is central to RLHF rather than an optional addition. - Preference collection challenges: State the two main challenges the introduction identifies for preference data collection, including what on-policy means here.
Why it matters: These constraints help explain why gathering useful preference data is difficult.
Key terms: preference data, proxy signal, loss functions, RLHF, operational complexity, cost of collection, on-policy
11.1 Why we need preference data
Explains why RLHF uses preference data to train reward models and why comparing answers is a practical way to collect it.
- Preference data for reward models: Explain why RLHF collects preference data to train reward models rather than trying to capture complex human values directly in a single reward function.
Why it matters: This explains what preference data contributes to RLHF without treating a reward model as a complete account of human values. - Comparisons over answer generation: Explain why distinguishing a good answer from a bad one makes preference data practical to collect.
Why it matters: The comparison task helps explain why preference data is a useful source of supervision.
Key terms: preference data, reward models, reward function, good answer, bad answer, supervising data collection
11.2 Collecting preference data
This section explains the practical difficulty of collecting human preference data, why model-related data matters, and what a chosen answer in a comparison means.
- Human data collection burden: Explain why teams may avoid collecting human preference data from scratch and use other sources instead.
Why it matters: The source and cost of preference data shape what an RLHF team can realistically build. - On-policy preference data: Describe what on-policy means for preference data and why collecting it from related model checkpoints helps RLHF training.
Why it matters: Judgments on answers resembling the model's own outputs are especially useful for improving that model. - Relative meaning of chosen: Distinguish a chosen answer in preference data from an answer that is globally correct.
Why it matters: A preference label tells the model which response is better in a comparison, not that the selected response is flawless.
Key terms: human preference data, AI feedback data, off-the-shelf reward models, on-policy, checkpoint, current family of models, supervised fine-tuning, chosen answer, preference data, alternatives
11.2.1 Interfaces
This section shows how interfaces shape preference-data collection, from dedicated answer comparisons to feedback gathered during everyday use.
- Interface effects on judgments: Explain why the collection interface matters and why its effects are hard to predict, using response speed as the section’s example.
Why it matters: A preference judgment can reflect the experience of using the interface as well as the model’s answer. - Paired-answer collection settings: Describe how an interface collects a preference between two answers, and distinguish dedicated labeling from comparisons shown to everyday users.
Why it matters: The same kind of answer comparison can be gathered in different settings and used for different purposes. - Other interface feedback signals: Describe the other feedback forms shown here: a tie in paired comparisons, positive or negative feedback on one response, and selection among generated images.
Why it matters: Recognizing different feedback controls helps identify what signal an interface actually collects.
Key terms: interface, user experience, preference data, data labeler, preference, training-data-only interface, evaluation, tie, positive or negative, thumbs-up
11.2.2 Rankings vs. ratings
This section distinguishes rankings from ratings and explains how individual scores and Likert-scale comparisons can produce preference data.
- Rankings and ratings distinction: Distinguish rankings, which order model completions relative to one another, from ratings, which score each piece of text. State that training commonly uses rankings, while ratings may serve as metadata.
Why it matters: The same-looking numerical value can represent different kinds of preference data. - Rating-derived preference pairs: Describe how scores for multiple completions to one prompt can be turned into a preference pair by choosing the highest-rated completion and pairing it with a randomly selected lower-scored one.
Why it matters: This shows how independently scored responses can supply comparative preference data. - Likert-scale relative preferences: Explain how a Likert scale records which response is preferred, and why its integer value does not make it an individual rating. Describe the section’s even-scale example and its eventual reduction to a binary reward-model signal.
Why it matters: The recorded number’s meaning—and what is retained for training—depends on how the comparison was collected.
Key terms: rankings, ratings, model completions, metadata, preference data, highest-rated completion, lower-scored completion, Likert scale, relative rankings, ties
11.2.3 Multiturn data
This section explains how multiturn preference data is collected and used for training, and why real-time conversations make collection difficult.
- Multiturn Preference Collection: Describe where preferences may be collected in a multiturn conversation and how the conversation proceeds when every response is judged.
Why it matters: Where judgments are collected determines which response becomes the next turn of the conversation. - Conversation Turns as Training Prompts: Explain how turns from a multiturn conversation can be used in training and what cautions the section gives.
Why it matters: A single conversation can supply multiple training prompts, but preparing them carelessly can bias the data. - Real-Time Labeling Constraints: Explain why separating prompt creation from preference labeling makes multiturn collection difficult, and identify the open question about who should label.
Why it matters: Who creates prompts and labels responses affects whether a real-time multiturn collection process is practical.
Key terms: multiturn data, final prompt, chosen answer, training prompts, unroll, masked from the loss, prompt creator, labeling the preference, sycophancy, real time
11.2.4 Structured preference data
This section explains how checkable properties of responses can create preference data automatically, then notes other forms of feedback that RLHF can use.
- Automatically checkable preferences: Explain how a domain with checkable outcomes can supply chosen and rejected responses without a subjective quality judgment.
Why it matters: It shows why some preference data can be constructed from the task itself. - Constraint-based comparison construction: Describe how the poem example constructs a preference pair and why the comparison prompt must retain the constraint.
Why it matters: The constraint is what makes the comparison meaningful as instruction-following data. - Alternative feedback signals: Distinguish the other feedback forms mentioned from paired preference data and describe their stated tradeoff.
Why it matters: It keeps paired chosen-versus-rejected examples from seeming like the only possible feedback format.
Key terms: structured preference data, chosen response, rejected response, constraint, synthetic preference data, chosen column, directional labels, fine-grained feedback, natural language feedback
11.2.5 Sourcing and contracts
This section explains the practical constraints of buying human preference data and why a vendor contract works best as an iterative, time-sensitive part of model training.
- Vendor access and contracts: Explain why obtaining preference data from a vendor can be slow or uncertain, and identify contractual terms a buyer needs to check.
Why it matters: A data plan must account for access, delivery, and usage rights—not just the number of labels ordered. - Task and schedule planning: Describe what the buyer and provider must settle before collection, and why changing collection dates may be difficult.
Why it matters: Instructions and scheduling determine what annotators can deliver and when the training team can use it. - Iterative batch collection: Trace how weekly deliveries, early calibration, and training feedback are meant to improve preference data during a contract.
Why it matters: The value of a batch depends partly on whether the team can learn from it while later batches can still be adjusted. - Instruction data time pressure: Distinguish the time pressure of collecting preference data from that of buying human-written instruction data.
Why it matters: The urgency described for preference-data contracts should not be assumed to apply equally to every kind of human training data.
Key terms: data vendors, supply-limited, contracts, data buyer, data provider, instructions, schedule, weekly batches, calibration, annotation guidelines
11.3 Bias: Things to watch out for in data collection
The section explains how subtle biases in collected preference data can reach the final model and why some are harder to address than others.
- Bias in preference data: Explain how bias in preference data can affect the final model, using prefix bias as an example.
Why it matters: A model trained on biased preferences may reflect those biases. - Uneven difficulty of mitigation: Distinguish the section’s examples of subtle biases and explain why some are harder to detect and mitigate than others.
Why it matters: Noticing which biases are easy to miss helps explain why collecting strong preference data takes more than giving labelers criteria.
Key terms: preference data, prefix bias, sycophancy, verboseness, formatting habits, labeling criteria
11.4 Open questions in RLHF preference data
The section explains why human preference data raises unresolved questions about collection, representation, and whether trained models reflect the preferences intended by annotators.
- Sparse comparisons and broad preferences: Explain why preference-data curation and its impacts deserve scrutiny: individual comparisons between two texts provide limited evidence for the broad, complex preferences RLHF aims to capture.
Why it matters: A simple preference comparison can carry more weight in training than the interaction alone appears to justify. - Collection and representation questions: Describe the open questions about whether collection setting, feedback format, and annotator population shape the preferences recorded and their effects on models.
Why it matters: Who gives feedback, under what conditions, and in what form may affect what the resulting data represents. - Annotator intent and model behavior: Explain the unresolved question of whether trained model behavior matches the preferences or specifications given to annotators, and why the section says this is hard to check.
Why it matters: Stating what annotators should prefer is not the same as being able to verify that models express those preferences.
Key terms: preference data, pairwise preferences, data collection contexts, multiaxis feedback mechanisms, population demographics, preference annotator disagreement, preference annotators, Model Spec, audit, specification
11.5 Summary
Where this is used later
- (chapter intro) (chapter 12) · Why RLHF needs preference data.
- Balancing AI and human feedback data (chapter 12) · The alternative of AI feedback provides context for balancing AI and human feedback.
- AI feedback (chapter 12) · Introduces collecting preference data.
- From offline to on-policy distillation (chapter 12) · The distinction between data from current models and other data provides context for on-policy distillation.
- The roles of synthetic data (chapter 12) · Provides an example of preference data created synthetically.
- Model specifications (chapter 17) · Provides context for the distinction between a stated model specification and observable model behavior
Concepts explained in this chapter
- preference dataA pairwise comparison is an evaluation that determines which of two items is preferred.
- on-policy methodOn-policy optimization, or on-policy learning, updates a policy using data generated by that same policy or its current update distribution.
- preference tuningPreference optimization is model training that adjusts behavior toward preferred outputs using preference-based objectives or rewards.
- data collectionData acquisition is the process of obtaining data from external providers or internal collection operations for a defined purpose.
- data providerA data vendor is an external provider that organizes people or processes to produce labeled or curated data for a customer.
- synthetic preference dataSynthetic preference data is preference data generated by models or algorithmic processes rather than directly collected from human judgment.
- reinforcement learning with human feedbackReinforcement learning from human feedback (RLHF) is a model-alignment technique that optimizes behavior using human preference signals, often through a learned reward model or direct preference objective.
- labeling guidelineAnnotation guidelines are documented instructions that specify how annotators should perform a data-labeling task, including priorities and decision criteria.
- fine-tuningFine-tuning is further training of a pretrained model on a task-specific or behavior-specific objective.
- instruction dataAn instruction dataset is a collection of prompts or instructions paired with target responses for supervised model adaptation.
- AI feedbackAI feedback is evaluative information about model outputs or actions generated by an artificial-intelligence system rather than directly supplied by people.
- accuracyTraining accuracy is the proportion of training examples for which a model produces the correct output.
- policyA policy is a function that maps an environment state to a probability distribution over actions.
- post-trainingPost-training is the set of optimization and adaptation procedures applied to a pretrained model to improve its usefulness for downstream tasks or alignment objectives.
- proxy metricA proxy metric is an imperfect measurable quantity used to approximate a desired quality or objective.
- prefix biasPrefix bias is a bias in which the beginning of a model completion disproportionately influences a preference judgment.
- sycophancySycophancy is a model behavior that favors agreement or flattering responses over truthful or independently justified responses.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.