Preference data records comparative judgments between candidate outputs or behavior segments. A judge chooses A, B, or a tie; some labeling tasks also allow an incomparable verdict. Unlike a score for one answer, a preference expresses relative quality, not an absolute rating. section 4.4.2 section 7.4.2
Pairwise judgments are often easier and more reliable than assigning precise scores. They are useful for offline model comparisons and for training reward models in reinforcement learning from human feedback (RLHF). A full comparison of n candidates requires O(n²) evaluations, making it costly for large candidate sets or individual online responses. section 4.2.2 section 4.4.2
A practical preference-labeling workflow:
- Collect two candidate responses or two short segments of recorded behavior for the judge to compare. section 4.4.2 section 7.4.2
- Use a clear criterion to select the preferred candidate, or record a tie or incomparable result when the labeling scheme permits it. section 7.4.2 section 8.2.2
- For an LLM judge, repeat the comparison with candidate positions swapped; require agreement for a confident verdict. This doubles inference cost. section 4.4.2
- Accumulate labeled pairs for evaluation or reward-model training. In one RLHF workflow, a reward model learns predicted preferences from segment rewards and human labels using the Bradley-Terry model. section 4.4.2 section 7.4.2
Comparing financial answers
A judge comparing two responses to a financial question could follow the principle: choose the response that does not invent facts absent from the retrieved source. A separate principle could favor an explicit “I do not know” over a plausible guess when the answer is missing. section 8.2.2
Figure 7.5 The three concurrent processes in Christiano et al.'s 2017 system. The policy generates trajectory segments that feed a preference-labeling queue. Human annotators compare pairs and return binary judgments, which accumulate in the preference dataset. The reward predictor is trained on the dataset, and its scores drive the policy-gradient updates. Arrows show the data flow; all three loops run continuously rather than in sequence.
Follow the flow from policy-generated segments through human preference labels to reward-model training and policy updates. section 7.4.2
Preference labels reflect the rubric, the labelers, and the candidates they see. Ambiguous rubrics and unrepresentative labeler pools can introduce noise or bias; comparisons also cannot establish preferences in regions they never covered. AI labelers can reduce annotation demands, but need periodic calibration against fresh human labels as policy outputs change. section 7.6.2 section 8.1.1 section 8.3.3



