liveBook · Manning

Preference pair

As explained in LLM Customization and Fine-Tuning

A training example containing two responses to the same prompt and a label indicating which is preferred.

Two IT support answers can both be correct, yet only one may follow the required format and point to an approved tool. A preference pair captures that distinction: a prompt, a chosen response and a rejected response to the same prompt. The rejected response is less preferred, not necessarily incorrect. section 8.1 section 8.2.1

Preference optimization uses pairs to refine an already supervised fine-tuned model. Direct Preference Optimization (DPO) trains the model to favor the chosen response over the rejected one relative to a frozen reference checkpoint, without a separate reward model or reinforcement-learning loop. section 8.1 section 8.2

Choosing an IT support answer

For a development-tool installation question, the chosen answer gives a direct opening, steps using the internal Software Center and an escalation line. The rejected answer gives a technically correct public-download walkthrough, but misses the organization’s preferred process. section 8.2.1

Table 8.1 Preference pair format (one record from train.jsonl)
Field
Content
prompt
[{"role": "system", "content": "You are an IT support..."}, {"role": "user", "content": "What is the terminal command to open the default web browser?"}]
chosen
[{"role": "assistant", "content": "`sensible-browser` is the command you are looking for, or `xdg-open <URL>`."}]
rejected
[{"role": "assistant", "content": "On Raspberry Pi Ubuntu, I opened a webpage fullscreen in kiosk mode at startup with `chromium-browser --app=<URL> --kiosk &` (a narrow, off-topic recipe)."}]

The record shows the prompt, chosen and rejected fields as chat-message lists. section 8.2.1

Build and use pairs with the preference signal in mind:

  • Use human judgments where available. In one IT dataset, higher- and lower-scoring Stack Exchange answers to the same question supply chosen and rejected responses. section 8.2.1
  • A cheaper alternative pairs an SFT model’s answer with a weaker base model’s answer. It can target style and verbosity, but reflects the model gap rather than human judgment. section 8.2.1
  • Exclude evaluation prompts and malformed pairs during preparation; keep validation pairs separate from training. section 8.2.1
  • Prioritize clean, consistent comparisons. A 300-pair run demonstrates the technique; production guidance suggests 1,000–5,000 pairs, with quality more important than quantity. section 8.8.3

Pairs are not a substitute for a capable starting model or good supervised data. For factual errors, improve supervised data or retrieval first; factuality-focused pairs require consistent labels and separate factuality evaluation. section 8.8.3 section 8.8.1

How LLM Customization and Fine-Tuning covers it

How deep it goes
Explains pair construction, chat-message formatting, data splitting and the DPO training objective, with practical quality and scale guidance.
Examples
Uses community-ranked IT support answers and a development-tool installation comparison to illustrate preferred and rejected responses.
What it assumes
Assumes a supervised fine-tuned checkpoint before preference optimization.
What you can do after
Apply preference pairs using TRL’s DPOTrainer to refine a supervised fine-tuned model.
How it connects
Connects preference pairs to supervised fine-tuning, direct preference optimization, reward models and reinforcement learning from human feedback.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.