Preference data
Preference data is a collection of judgments about the relative preference or quality of alternatives, often candidate outputs for the same input.
Reinforcement Learning from Human Feedback
A pairwise comparison is an evaluation that determines which of two items is preferred.
- How it is used
- Provides a proxy preference signal for reward modeling and preference fine-tuning when desired behaviors resist precise reward functions.
- How deep it goes
- Develops pairwise collection formats and grouped reward-model loss, then discusses sourcing, on-policy sampling, and bias.
- Examples
- Uses relative ranking scales, correctness-based pairs, and a goldfish-poem instruction constraint to illustrate preference construction.
- What you can do after
- Evaluate and understand preference-data formats, collection choices, and biases when planning a preference-training pipeline.
Read how Reinforcement Learning from Human Feedback explains preference data →
LLM Evaluation and Alignment, The Foundational Ideas
Pairwise comparison is an evaluation method in which a judge chooses the preferred option from two candidate outputs, possibly allowing a tie.
- How it is used
- Uses pairwise judgments for offline model comparison and as training data for RLHF reward models.
- How deep it goes
- Explains labeling choices, reward-model training, position-bias mitigation, scaling costs, and annotation limitations.
- Examples
- Illustrates labels with trajectory clips and principles for comparing financial research answers.
- What you can do after
- Evaluate and understand preference judgments, their use in reward modeling, and their limitations.
Read how LLM Evaluation and Alignment, The Foundational Ideas explains preference data →
Machine Learning System Design
Pairwise comparison is an evaluation technique that asks a rater to compare two alternatives rather than assign an absolute score to one alternative.
- How it is used
- Frames pairwise comparison as a labeling design for evaluating alternatives such as chatbot responses.
- How deep it goes
- Explains ternary judgments and variance reduction, then qualifies majority voting without deriving a broader preference model.
- Examples
- Uses chatbot answers, correctness and tone criteria, ternary A-or-B judgments, and three labelers with 70% individual accuracy.
- What you can do after
- Evaluate alternatives with lower-variance ternary comparisons while checking whether labelers’ errors are correlated.
Read how Machine Learning System Design explains preference data →
LLM Customization and Fine-Tuning
A collection of judgments or comparisons expressing preferences between candidate model outputs.
- How it is used
- Supplies comparative response judgments for preference optimization and reward-model training.
- How deep it goes
- Compares four data sources by cost and signal quality, then traces prepared pairs through training and evaluation.
- Examples
- Uses community-scored Stack Exchange answers and 300 human-graded IT support pairs to illustrate preference data.
- What you can do after
- Evaluate preference-data sources against annotation cost, signal quality, and deployment context.
Read how LLM Customization and Fine-Tuning explains preference data →
Also covered in
- Generative AI in Action
- Introduction to Generative AI, Second Edition
- Build an AI Agent (From Scratch)
- Grokking AI Algorithms, Second Edition
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- reward modelA reward model is a learned model that assigns scores to outputs, actions, or outcomes to represent their desirability according to preferences or evaluation criteria.
- RLHFReinforcement learning from human feedback is a model-training approach that uses human evaluations or preferences as feedback to optimize model behavior.
- reinforcement learningReinforcement learning is a machine-learning paradigm in which an agent learns behavior or a decision policy through interaction with an environment and evaluative feedback, such as rewards.
Related concepts
- reward modelused by · A reward model is a learned model that assigns scores to outputs, actions, or outcomes to represent their desirability according to preferences or evaluation criteria.
- Direct Preference Optimizationused by · Direct Preference Optimization is a technique for training a model directly from preference data, such as preferred and dispreferred outputs, without separately fitting a reward model.
- RLHFused by · Reinforcement learning from human feedback is a model-training approach that uses human evaluations or preferences as feedback to optimize model behavior.
- Chatbot Arenaused by · Chatbot Arena is an online evaluation platform and leaderboard that compares conversational language models through users’ pairwise preference judgments of their responses.
- datasetkind of · A dataset is an organized collection of related data items, records, observations, or examples treated as a unit for storage, processing, or analysis.
- direct alignment algorithmused by · A direct alignment algorithm optimizes a model directly from preference data without learning an intermediate reward model or using a reinforcement-learning optimizer.
Questions about preference data
What is preference data?
Preference data is a collection of judgments about the relative preference or quality of alternatives, often candidate outputs for the same input.
Which Manning books explain preference data?
Reinforcement Learning from Human Feedback, LLM Evaluation and Alignment, The Foundational Ideas, Machine Learning System Design and LLM Customization and Fine-Tuning.
What should I know before learning preference data?
Manning books that teach preference data usually introduce language model, reward model, RLHF and reinforcement learning first.
What is preference data a kind of?
Preference data is a kind of dataset. A dataset is an organized collection of related data items, records, observations, or examples treated as a unit for storage, processing, or analysis.
What is preference data also called?
Preference data is also called pairwise preferences or preference dataset.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.



