liveBook · Manning

Direct Preference Optimization

As explained in LLM Customization and Fine-Tuning

A method that tunes a model directly from preference comparisons without separately training a reward model.

Two help-desk answers can both be correct, yet only one follows an organization’s preferred format and points users to its approved tools. Direct Preference Optimization (DPO) tunes a model on such comparisons, teaching it to prefer one response over another without separately training a reward model. section 8.2 section 8.2.1

DPO typically refines a model that has already undergone supervised fine-tuning (SFT). Each example contains a prompt, a chosen response, and a rejected response. The objective favors the chosen response over the rejected one relative to a frozen reference checkpoint. The policy model is updated; the reference stays fixed to constrain drift. Unlike a conventional RLHF pipeline, DPO needs neither a separate reward model nor an RL training loop. section 8.2 section 8.2.5

Choosing a preference signal

For a Windows network-adapter reset, a concise chosen answer gives four steps and stops. The rejected answer provides the same steps but adds an introduction, a restated question, and a sign-off. This pair isolates conciseness rather than correctness. section 8.2.8

Table 8.6 Concise (chosen) versus verbose (rejected) responses to the same prompt
Concise response (chosen)
Verbose response (rejected)
Summary: Reset the adapter from Windows Settings. Steps: (1) Open Settings > Network & Internet > Advanced network settings; (2) Select Network reset; (3) Click Reset now and confirm; (4) Restart when prompted.
Resetting your network adapter is a common fix for connectivity issues, and Windows makes it straightforward. To reset your network adapter on Windows, follow these steps: (1) Open Settings > Network & Internet > Advanced network settings; (2) Select Network reset; (3) Click Reset now and confirm; (4) Restart when prompted. Let me know if you need anything else or if you run into trouble.

Compare the chosen and rejected responses to see how a pair isolates one preference. section 8.2.8

A practical DPO workflow:

  • Prepare representative preference pairs and exclude evaluation prompts from training. Human-graded pairs reflect human judgments; stronger-versus-weaker model pairs are cheaper but reflect the gap between models instead. section 8.2.1
  • Train from a solid SFT checkpoint using a frozen reference. A worked TRL DPOTrainer run uses a learning rate of 5e-6, beta of 0.1, and one epoch over 270 training pairs. section 8.2.3 section 8.8.3
  • Compare base, SFT, and DPO models on held-out prompts, measuring the specific preference being tuned as well as correctness. Test safety regressions separately after preference tuning. section 8.2.6 section 8.2.8

Beta controls how closely training stays to the reference: values below 0.05 risk excessive drift and mode collapse, while values above 0.3 may produce little change. DPO also holds policy and reference models in memory. In one full-parameter 4B run, peak use was about 54 GB across three 24 GB GPUs; gradient checkpointing reduced memory at the cost of extra compute. section 8.2.3

In one IT-support evaluation, SFT and DPO had nearly identical held-out Token-F1 scores: 0.168 and 0.166. A separate conciseness run produced answers 21% shorter, with a blinded judge preferring DPO in 27 of 44 non-tied comparisons. These results illustrate why the evaluation must match the preference axis; a small test set also makes minor differences noisy. section 8.2.7 section 8.2.8

Check SFT data quality before adding DPO. Use DPO for a clearly stated quality or style preference; consider GRPO or reinforcement fine-tuning when rewards are verifiable, and RLHF when finer-grained preference generalization warrants its added complexity. DPO does not automatically fix factual errors: improve SFT data or retrieval, or explicitly train and evaluate for factuality. Monitor for length bias, likelihood displacement, and safety regressions. section 8.7 section 8.8.3 section 8.10

How LLM Customization and Fine-Tuning covers it

How deep it goes
Explains the policy-reference objective and walks through preference preparation, training, evaluation, and safety testing.
Examples
Uses IT-support preference pairs and a Windows network-reset example to demonstrate concise versus verbose responses.
What it assumes
Assumes a capable supervised fine-tuned checkpoint and preference pairs, and shows how to format and split the pairs.
What you can do after
Apply DPO via TRL’s DPOTrainer and evaluate preference gains, correctness, and safety against the starting model.
How it connects
Connects DPO with supervised fine-tuning, reinforcement learning from human feedback, GRPO, and reinforcement fine-tuning.

Also explained in

Compare how 3 books explain Direct Preference Optimization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.