Preference data records judgments about which model output is better for a prompt. It supplies a practical proxy when desired human behavior cannot be written as a precise reward function, making it the central training signal for preference fine-tuning and reinforcement learning from human feedback (RLHF). The common unit is a pairwise comparison: a chosen completion and a rejected completion. section 11.0 section 11.5
Comparing two answers is often easier than generating an ideal answer from scratch. A reward model can learn from these relations to score new prompt–completion pairs, then provide the scalar signal used during RLHF optimization. Preference data therefore translates judgments about answer quality into a form training pipelines can consume. section 11.1 section 1.3
A useful preference-data pipeline makes several design choices:
- Generate completions from the current model family when possible. This on-policy data better covers the distributions the model will actually produce during optimization. section 11.2
- Choose rankings or ratings. Rankings directly express relative order and are common for training; ratings can serve as metadata or can be converted into chosen–rejected pairs. section 11.2.2
- Group comparisons from the same prompt during reward-model training. All pairs from prompts with many completions are correlated, so unweighted shuffling can overemphasize those prompts. section 5.3.2
- Treat collection as an iterative operational process involving annotation instructions, calibration, vendors or annotators, quality loss in early batches, and integration with training. section 11.2.5
- Audit for biases that may transfer into the trained model, including prefix bias, sycophancy, verboseness, and formatting preferences. section 11.3
Turning an instruction constraint into a preference pair
Structured tasks can produce synthetic preference data without asking a rater for a subjective quality judgment. For a prompt requiring every sentence of a goldfish poem to start with “g,” a completion that follows the constraint can be marked chosen over one generated without it. The stored prompt must include the constraint; otherwise the comparison loses the condition that defines preference. section 11.2.4
Human data is costly and difficult to collect, so teams may use AI feedback, off-the-shelf reward models, or synthetic data. Synthetic pairwise data can be created by having a model judge two completions under written principles, but model judges introduce their own trade-offs and biases. A mixed strategy can route difficult examples to humans while using cheaper AI feedback at larger scale. section 11.2 section 12.4.1 section 12.5
Table 11.1 An example 5-point Likert scale between two responses, A and B
| A\(>>\)B | A\(>\)B | Tie | B\(>\)A | B\(>>\)A |
| 1 | 2 | 3 | 4 | 5 |
Use the five-point relative-ranking scale to distinguish strong and weak preferences from ties before deciding whether training will reduce labels to a binary chosen–rejected signal. section 11.2.2
