A human preference is a comparative judgment that one output or behavior is more desirable than another. Comparing two examples can be easier than specifying a reward by hand or demonstrating the desired behavior from scratch. section 7.4.2
Preference labels can turn human judgment into a training signal. A policy generates behavior, annotators compare pairs of short segments, and a reward model learns to predict their judgments. The model’s predicted rewards then guide policy updates. These processes run concurrently rather than as a single pass. section 7.4.2
Figure 7.5 The three concurrent processes in Christiano et al.'s 2017 system. The policy generates trajectory segments that feed a preference-labeling queue. Human annotators compare pairs and return binary judgments, which accumulate in the preference dataset. The reward predictor is trained on the dataset, and its scores drive the policy-gradient updates. Arrows show the data flow; all three loops run continuously rather than in sequence.
Follow how policy rollouts, human comparisons, and reward-model predictions feed one another. section 7.4.2
A preference-learning loop has three connected steps:
- Record trajectories from the policy and sample pairs of short segments. section 7.4.2
- Ask annotators to choose the better segment, report a tie, or mark the pair incomparable. section 7.4.2
- Train a reward model on the accumulated comparisons. The Bradley–Terry model relates predicted cumulative rewards to pairwise preferences, and cross-entropy measures prediction error. section 7.4.2
Preferences can encode an unwanted trade-off
If labelers reward harmlessness without penalizing evasion, they may prefer evasive refusals. A reward model can learn that preference, making the resulting policy less helpful than intended. Inspect the rubric and the judgments, not just the resulting score. section 8.6 section 7.5.2
Human preferences are noisy judgments aggregated across individuals. One reported ranking task had roughly 73% inter-labeler agreement; screening and calibration were used to improve label quality. A preference label also gives little information about why one response is better. section 7.5.2 section 8.6
AI-generated preferences guided by written principles can reduce dependence on human labeler capacity, but they introduce different risks. Principles may leave gaps, and an AI labeler’s own style can leak into judgments. Sanity checks with another labeler and explicit style principles can help detect or counter that leakage. section 8.6 section 8.3.3





