liveBook · Manning

Preference data

As explained in LLM Customization and Fine-Tuning

A collection of judgments or comparisons expressing preferences between candidate model outputs.

Preference data records judgments about which of two candidate responses is better for the same prompt. It gives preference optimization a signal for qualities such as conciseness, tone, and formatting—not just objective correctness. section 8.2.2 section 8.10

Direct preference optimization (DPO) trains on preference pairs without a separate reward model. In reinforcement learning from human feedback (RLHF), human preferences instead train a reward model, which supplies the signal for policy optimization. section 8.10 section 8.4

Choose a data source by balancing cost, throughput, and the judgment needed:

  • Model-model pairs label the stronger model’s response as chosen. They are inexpensive and consistent for style and formatting, but reflect the gap between models rather than nuanced human judgment. section 8.2.2
  • Human-model pairs use a response written or edited by a person as the chosen answer. They offer high-fidelity signals for brand voice and subjective or regulated tasks, but skilled annotation is costly and slow; selecting or lightly editing generated candidates can cost less. section 8.2.2
  • Log-derived pairs use signals such as accepted versus re-asked answers or resolved versus escalated conversations. Existing feedback loops can supply them continuously at no marginal annotation cost, but the signals are noisy. section 8.2.2
  • Judge-model pairs turn a stronger evaluator’s scores or rankings into preferences. They scale better than human annotation, at a per-pair cost above model-model or log-derived pairs. section 8.2.2
Table 8.2 Preference data source comparison
Source
Cost per pair
Signal quality
Best for
Model-model
~$0 (compute only)
Moderate
Style, verboseness, formatting
Human-model
$1 to $5
High
Brand voice, regulated content
Log-derived
~$0 (existing infra)
Noisy
Continuous improvement at scale
Judge-model
$0.01 to $0.10
Moderate-high
Scaling when humans are too slow

Compare indicative costs, signal quality, and best-fit uses across the four sources. section 8.2.2

IT support preference pipeline

A DPO run used 300 human-graded IT support pairs, split into 270 training and 30 validation pairs after a contamination guard. Training ran for one epoch from an SFT checkpoint. Token-F1 was essentially unchanged, from 0.168 to 0.166; a conciseness-tuned model produced answers 21% shorter, preferred 61% of the time by a blinded judge. Safety testing found no regression relative to SFT, although both passed only one of four checks. section 8.2.6 section 8.10

For RLHF, annotation reliability is a separate cost: typical inter-annotator disagreement is reported at 20–30%, requiring multiple annotations per pair for reliable labels. Monitor preference-optimized models for length and verboseness bias, and test safety after optimization. section 8.4.1 section 8.10

How LLM Customization and Fine-Tuning covers it

How deep it goes
Compares four data sources by cost and signal quality, then traces prepared pairs through training and evaluation.
Examples
Uses community-scored Stack Exchange answers and 300 human-graded IT support pairs to illustrate preference data.
What you can do after
Evaluate preference-data sources against annotation cost, signal quality, and deployment context.
How it connects
Connects preference data to DPO, RLHF, reward models, supervised fine-tuning, and safety regression testing.

Also explained in

Compare how 4 books explain preference data →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.