A prompt is an input to a model and may combine system instructions with a user query. Treat it as a production artifact: a high score on design examples can reflect overfitting and may not carry over to real inputs. Check it on held-out data, across repeated runs, and for brittleness before deployment. section 4.4 section 8.2.1
For prompts used to generate distillation data, prioritize coverage over raw volume:
- Define a narrow task; a specific IT-support scope is easier to distill than a general-purpose assistant. section 7.1.1
- Sample varied phrasings, difficulty levels, and edge cases that reflect production queries. Expand real queries with paraphrases; without query logs, gather seed questions from domain experts and generate variations. section 7.1.1
Prompt in a preference pair
A DPO record can represent the prompt as a message list containing a system instruction and a user question, alongside chosen and rejected assistant responses. For example, the question about opening the default browser is paired with a direct command as the chosen answer and an off-topic kiosk-mode recipe as the rejected answer. section 8.2.1
Listing 8.1 prepare_preference_data.py (key sections)
# Key sections of code/chapter08/scripts/prepare_preference_data.py
SYSTEM_PROMPT = ("You are an IT support assistant. "
"Provide clear, step-by-step answers.")
def main():
args = parse_args()
raw = list(read_jsonl(args.preferences)) #A
eval_prompts = _eval_prompts(args.eval_valid)
pairs = []
for rec in raw:
prompt = rec["prompt"].strip()
chosen = rec["chosen"].strip() #B
rejected = rec["rejected"].strip() #C
if prompt in eval_prompts:
continue
pairs.append({
"prompt": [{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": prompt}],
"chosen": [{"role": "assistant", "content": chosen}],
"rejected": [{"role": "assistant", "content": rejected}],
})
rng = random.Random(args.seed); rng.shuffle(pairs) #D
valid_pairs = pairs[:args.num_valid]
train_pairs = pairs[args.num_valid:]
write_jsonl(out_dir / "train.jsonl", train_pairs)
write_jsonl(out_dir / "valid.jsonl", valid_pairs)Inspect the preparation script’s message-list structure for preference data. section 8.2.1































































