liveBook · Manning

Preference data

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Pairwise comparison is an evaluation method in which a judge chooses the preferred option from two candidate outputs, possibly allowing a tie.

Preference data records comparative judgments between candidate outputs or behavior segments. A judge chooses A, B, or a tie; some labeling tasks also allow an incomparable verdict. Unlike a score for one answer, a preference expresses relative quality, not an absolute rating. section 4.4.2 section 7.4.2

Pairwise judgments are often easier and more reliable than assigning precise scores. They are useful for offline model comparisons and for training reward models in reinforcement learning from human feedback (RLHF). A full comparison of n candidates requires O(n²) evaluations, making it costly for large candidate sets or individual online responses. section 4.2.2 section 4.4.2

A practical preference-labeling workflow:

  • Collect two candidate responses or two short segments of recorded behavior for the judge to compare. section 4.4.2 section 7.4.2
  • Use a clear criterion to select the preferred candidate, or record a tie or incomparable result when the labeling scheme permits it. section 7.4.2 section 8.2.2
  • For an LLM judge, repeat the comparison with candidate positions swapped; require agreement for a confident verdict. This doubles inference cost. section 4.4.2
  • Accumulate labeled pairs for evaluation or reward-model training. In one RLHF workflow, a reward model learns predicted preferences from segment rewards and human labels using the Bradley-Terry model. section 4.4.2 section 7.4.2

Comparing financial answers

A judge comparing two responses to a financial question could follow the principle: choose the response that does not invent facts absent from the retrieved source. A separate principle could favor an explicit “I do not know” over a plausible guess when the answer is missing. section 8.2.2

Figure 7.5 The three concurrent processes in Christiano et al.'s 2017 system. The policy generates trajectory segments that feed a preference-labeling queue. Human annotators compare pairs and return binary judgments, which accumulate in the preference dataset. The reward predictor is trained on the dataset, and its scores drive the policy-gradient updates. Arrows show the data flow; all three loops run continuously rather than in sequence.

Follow the flow from policy-generated segments through human preference labels to reward-model training and policy updates. section 7.4.2

Preference labels reflect the rubric, the labelers, and the candidates they see. Ambiguous rubrics and unrepresentative labeler pools can introduce noise or bias; comparisons also cannot establish preferences in regions they never covered. AI labelers can reduce annotation demands, but need periodic calibration against fresh human labels as policy outputs change. section 7.6.2 section 8.1.1 section 8.3.3

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains labeling choices, reward-model training, position-bias mitigation, scaling costs, and annotation limitations.
Examples
Illustrates labels with trajectory clips and principles for comparing financial research answers.
What you can do after
Evaluate and understand preference judgments, their use in reward modeling, and their limitations.
How it connects
Connects pairwise comparison with RLHF, reward models, Bradley-Terry modeling, and Constitutional AI.

Also explained in

Compare how 4 books explain preference data →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.