Two IT support answers can both be correct, yet only one may follow the required format and point to an approved tool. A preference pair captures that distinction: a prompt, a chosen response and a rejected response to the same prompt. The rejected response is less preferred, not necessarily incorrect. section 8.1 section 8.2.1
Preference optimization uses pairs to refine an already supervised fine-tuned model. Direct Preference Optimization (DPO) trains the model to favor the chosen response over the rejected one relative to a frozen reference checkpoint, without a separate reward model or reinforcement-learning loop. section 8.1 section 8.2
Choosing an IT support answer
For a development-tool installation question, the chosen answer gives a direct opening, steps using the internal Software Center and an escalation line. The rejected answer gives a technically correct public-download walkthrough, but misses the organization’s preferred process. section 8.2.1
Table 8.1 Preference pair format (one record from train.jsonl)
|
Field
|
Content
|
|
prompt
|
[{"role":
"system", "content": "You are an IT support..."},
{"role": "user", "content": "What is the terminal
command to open the default web browser?"}]
|
|
chosen
|
[{"role":
"assistant", "content": "`sensible-browser` is the command
you are looking for, or `xdg-open <URL>`."}]
|
|
rejected
|
[{"role":
"assistant", "content": "On Raspberry Pi Ubuntu, I opened
a webpage fullscreen in kiosk mode at startup with `chromium-browser --app=<URL>
--kiosk &` (a narrow, off-topic recipe)."}]
|
The record shows the prompt, chosen and rejected fields as chat-message lists. section 8.2.1
Build and use pairs with the preference signal in mind:
- Use human judgments where available. In one IT dataset, higher- and lower-scoring Stack Exchange answers to the same question supply chosen and rejected responses. section 8.2.1
- A cheaper alternative pairs an SFT model’s answer with a weaker base model’s answer. It can target style and verbosity, but reflects the model gap rather than human judgment. section 8.2.1
- Exclude evaluation prompts and malformed pairs during preparation; keep validation pairs separate from training. section 8.2.1
- Prioritize clean, consistent comparisons. A 300-pair run demonstrates the technique; production guidance suggests 1,000–5,000 pairs, with quality more important than quantity. section 8.8.3
Pairs are not a substitute for a capable starting model or good supervised data. For factual errors, improve supervised data or retrieval first; factuality-focused pairs require consistent labels and separate factuality evaluation. section 8.8.3 section 8.8.1
