Preference data
Preference data is a collection of judgments or comparisons indicating the relative preference or quality of alternatives, often model outputs.
Reinforcement Learning from Human Feedback
A pairwise comparison is an evaluation that determines which of two items is preferred.
- How it is used
- Supplies proxy judgments for reward modeling, preference fine-tuning, and RLHF when desired behavior lacks a precise reward function.
- How deep it goes
- Details formats, on-policy sourcing, collection operations, and bias, while leaving pairwise adequacy and annotator demographics as open questions.
- Examples
- Uses Likert rankings, correct-versus-incorrect mathematical answers, constrained goldfish poems, and AI-judged completion pairs.
- What you can do after
- Operate/configure preference-data collection by selecting label formats, seeking on-policy generations, and monitoring transferable biases.
Read how Reinforcement Learning from Human Feedback explains preference data →
Machine Learning System Design
Pairwise comparison is an evaluation technique that asks a rater to compare two alternatives rather than assign an absolute score to one alternative.
- How it is used
- Frames pairwise comparison as a labeling design for evaluating alternatives such as chatbot responses.
- How deep it goes
- Explains ternary judgments and variance reduction, then qualifies majority voting without deriving a broader preference model.
- Examples
- Uses chatbot answers, correctness and tone criteria, ternary A-or-B judgments, and three labelers with 70% individual accuracy.
- What you can do after
- Evaluate alternatives with lower-variance ternary comparisons while checking whether labelers’ errors are correlated.
Read how Machine Learning System Design explains preference data →
Related concepts
- reward modelused by · A reward model is a learned model that assigns scores to outputs, actions, or outcomes to represent their desirability according to preferences or evaluation criteria.
- Direct Preference Optimizationused by · Direct Preference Optimization is a technique for training a model directly from preference data, such as preferred and dispreferred outputs, without separately fitting a reward model.
- reinforcement learning with human feedbackuses · Reinforcement learning with human feedback is a model-training approach that uses human evaluations or preferences to guide optimization of model behavior.
- preference tuninguses · Preference tuning is a model-training technique that adjusts a model’s behavior toward outputs favored by preference signals, judgments, objectives, or rewards.
- direct alignment algorithmused by · A direct alignment algorithm optimizes a model directly from preference data without learning an intermediate reward model or using a reinforcement-learning optimizer.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.