liveBook · Manning

Preference data

Also called pairwise preferences, preference dataset

Preference data is a collection of judgments about the relative preference or quality of alternatives, often candidate outputs for the same input.

How 4 Manning books explain it

Reinforcement Learning from Human Feedback

A pairwise comparison is an evaluation that determines which of two items is preferred.

How it is used
Provides a proxy preference signal for reward modeling and preference fine-tuning when desired behaviors resist precise reward functions.
How deep it goes
Develops pairwise collection formats and grouped reward-model loss, then discusses sourcing, on-policy sampling, and bias.
Examples
Uses relative ranking scales, correctness-based pairs, and a goldfish-poem instruction constraint to illustrate preference construction.
What you can do after
Evaluate and understand preference-data formats, collection choices, and biases when planning a preference-training pipeline.

Read how Reinforcement Learning from Human Feedback explains preference data →

LLM Evaluation and Alignment, The Foundational Ideas

Pairwise comparison is an evaluation method in which a judge chooses the preferred option from two candidate outputs, possibly allowing a tie.

How it is used
Uses pairwise judgments for offline model comparison and as training data for RLHF reward models.
How deep it goes
Explains labeling choices, reward-model training, position-bias mitigation, scaling costs, and annotation limitations.
Examples
Illustrates labels with trajectory clips and principles for comparing financial research answers.
What you can do after
Evaluate and understand preference judgments, their use in reward modeling, and their limitations.

Read how LLM Evaluation and Alignment, The Foundational Ideas explains preference data →

Machine Learning System Design

Pairwise comparison is an evaluation technique that asks a rater to compare two alternatives rather than assign an absolute score to one alternative.

How it is used
Frames pairwise comparison as a labeling design for evaluating alternatives such as chatbot responses.
How deep it goes
Explains ternary judgments and variance reduction, then qualifies majority voting without deriving a broader preference model.
Examples
Uses chatbot answers, correctness and tone criteria, ternary A-or-B judgments, and three labelers with 70% individual accuracy.
What you can do after
Evaluate alternatives with lower-variance ternary comparisons while checking whether labelers’ errors are correlated.

Read how Machine Learning System Design explains preference data →

LLM Customization and Fine-Tuning

A collection of judgments or comparisons expressing preferences between candidate model outputs.

How it is used
Supplies comparative response judgments for preference optimization and reward-model training.
How deep it goes
Compares four data sources by cost and signal quality, then traces prepared pairs through training and evaluation.
Examples
Uses community-scored Stack Exchange answers and 300 human-graded IT support pairs to illustrate preference data.
What you can do after
Evaluate preference-data sources against annotation cost, signal quality, and deployment context.

Read how LLM Customization and Fine-Tuning explains preference data →

Also covered in

What to know first

Related concepts

Questions about preference data

What is preference data?

Preference data is a collection of judgments about the relative preference or quality of alternatives, often candidate outputs for the same input.

Which Manning books explain preference data?

Reinforcement Learning from Human Feedback, LLM Evaluation and Alignment, The Foundational Ideas, Machine Learning System Design and LLM Customization and Fine-Tuning.

What should I know before learning preference data?

Manning books that teach preference data usually introduce language model, reward model, RLHF and reinforcement learning first.

What is preference data a kind of?

Preference data is a kind of dataset. A dataset is an organized collection of related data items, records, observations, or examples treated as a unit for storage, processing, or analysis.

What is preference data also called?

Preference data is also called pairwise preferences or preference dataset.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.