Preference data records judgments about which of two candidate responses is better for the same prompt. It gives preference optimization a signal for qualities such as conciseness, tone, and formatting—not just objective correctness. section 8.2.2 section 8.10
Direct preference optimization (DPO) trains on preference pairs without a separate reward model. In reinforcement learning from human feedback (RLHF), human preferences instead train a reward model, which supplies the signal for policy optimization. section 8.10 section 8.4
Choose a data source by balancing cost, throughput, and the judgment needed:
- Model-model pairs label the stronger model’s response as chosen. They are inexpensive and consistent for style and formatting, but reflect the gap between models rather than nuanced human judgment. section 8.2.2
- Human-model pairs use a response written or edited by a person as the chosen answer. They offer high-fidelity signals for brand voice and subjective or regulated tasks, but skilled annotation is costly and slow; selecting or lightly editing generated candidates can cost less. section 8.2.2
- Log-derived pairs use signals such as accepted versus re-asked answers or resolved versus escalated conversations. Existing feedback loops can supply them continuously at no marginal annotation cost, but the signals are noisy. section 8.2.2
- Judge-model pairs turn a stronger evaluator’s scores or rankings into preferences. They scale better than human annotation, at a per-pair cost above model-model or log-derived pairs. section 8.2.2
Table 8.2 Preference data source comparison
|
Source
|
Cost per pair
|
Signal quality
|
Best for
|
|
Model-model
|
~$0 (compute only)
|
Moderate
|
Style, verboseness, formatting
|
|
Human-model
|
$1 to $5
|
High
|
Brand voice, regulated content
|
|
Log-derived
|
~$0 (existing infra)
|
Noisy
|
Continuous improvement at scale
|
|
Judge-model
|
$0.01 to $0.10
|
Moderate-high
|
Scaling when humans are too slow
|
Compare indicative costs, signal quality, and best-fit uses across the four sources. section 8.2.2
IT support preference pipeline
A DPO run used 300 human-graded IT support pairs, split into 270 training and 30 validation pairs after a contamination guard. Training ran for one epoch from an SFT checkpoint. Token-F1 was essentially unchanged, from 0.168 to 0.166; a conciseness-tuned model produced answers 21% shorter, preferred 61% of the time by a blinded judge. Safety testing found no regression relative to SFT, although both passed only one of four checks. section 8.2.6 section 8.10
For RLHF, annotation reliability is a separate cost: typical inter-annotator disagreement is reported at 20–30%, requiring multiple annotations per pair for reliable labels. Monitor preference-optimized models for length and verboseness bias, and test safety after optimization. section 8.4.1 section 8.10



