Preference data

As explained in Machine Learning System Design

Pairwise comparison is an evaluation technique that asks a rater to compare two alternatives rather than assign an absolute score to one alternative.

Preference data records judgments by comparing alternatives instead of assigning each alternative an absolute score. For chatbot responses, labelers can choose whether version A is better, version B is better, or both are equal. section 6.2.4

Use pairwise preference labels when absolute ratings create avoidable disagreement:

• They are easier for labelers to collect than single-number scores and can reduce variance between workers. section 6.2.4
• Multiple criteria, such as factual correctness and tone of voice, can make judgments more consistent before comparison. section 6.2.4
• Aggregating several judgments can improve decision accuracy, but correlated errors and dependent labelers weaken the simple majority-vote assumption. section 6.2.4

Figure 6.2 Various approaches to data labeling

Compare alternative labeling designs, including absolute ratings, criteria-based labels, and pairwise comparisons. section 6.2.4

How Machine Learning System Design covers it

How deep it goes

Explains ternary judgments and variance reduction, then qualifies majority voting without deriving a broader preference model.

Examples

Uses chatbot answers, correctness and tone criteria, ternary A-or-B judgments, and three labelers with 70% individual accuracy.

What it assumes

Assumes familiarity with labeling and binary decisions while contrasting absolute scoring with pairwise judgments.

What you can do after

Evaluate alternatives with lower-variance ternary comparisons while checking whether labelers’ errors are correlated.

How it connects

Connects preference data to data labeling, chatbot evaluation, multiple evaluation criteria, and majority voting.

liveBookmanning.com

Read Machine Learning System Design on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Reinforcement Learning from Human FeedbackA pairwise comparison is an evaluation that determines which of two items is preferred.

Compare how 2 books explain preference data →

Related concepts

• reward modelused by · A reward model is a learned model that assigns scores to outputs, actions, or outcomes to represent their desirability according to preferences or evaluation criteria.
• Direct Preference Optimizationused by · Direct Preference Optimization is a technique for training a model directly from preference data, such as preferred and dispreferred outputs, without separately fitting a reward model.
• reinforcement learning with human feedbackuses · Reinforcement learning with human feedback is a model-training approach that uses human evaluations or preferences to guide optimization of model behavior.
• preference tuninguses · Preference tuning is a model-training technique that adjusts a model’s behavior toward outputs favored by preference signals, judgments, objectives, or rewards.
• direct alignment algorithmused by · A direct alignment algorithm optimizes a model directly from preference data without learning an intermediate reward model or using a reinforcement-learning optimizer.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning System Design. The section links point to the parts of the book it draws on.