Direct Preference Optimization
Direct Preference Optimization is a technique for training a model directly from preference data, such as preferred and dispreferred outputs, without separately fitting a reward model.
LLM Customization and Fine-Tuning
A method that tunes a model directly from preference comparisons without separately training a reward model.
- How it is used
- Uses DPO after supervised fine-tuning to refine subjective response qualities such as conciseness and organizational style.
- How deep it goes
- Explains the policy-reference objective and walks through preference preparation, training, evaluation, and safety testing.
- Examples
- Uses IT-support preference pairs and a Windows network-reset example to demonstrate concise versus verbose responses.
- What you can do after
- Apply DPO via TRL’s DPOTrainer and evaluate preference gains, correctness, and safety against the starting model.
Read how LLM Customization and Fine-Tuning explains Direct Preference Optimization →
Reinforcement Learning from Human Feedback
Direct preference optimization (DPO) directly trains a policy from pairwise preferred and dispreferred outputs relative to a reference policy.
- How it is used
- Uses pairwise preferences to align language models without a separate reward model or reinforcement learning optimizer.
- How deep it goes
- Derives the objective from the optimal KL-regularized policy and Bradley–Terry preferences, then details the loss, gradient, implementation, and weaknesses.
- Examples
- Uses a loss implementation and cites deployed models including Zephyr-Beta and Tülu 2.
- What you can do after
- Builds DPO from scratch using policy and reference log-probabilities, with implicit rewards for logging and optional reference-probability caching.
Read how Reinforcement Learning from Human Feedback explains Direct Preference Optimization →
Transformers in Action
Direct Preference Optimization is a preference-alignment technique that directly optimizes a language model from chosen and rejected responses without separately training an explicit reward model.
- How it is used
- Aligns language-model responses with human preferences by optimizing chosen over rejected outputs.
- How deep it goes
- Explains the preference-training workflow and trainer configuration, then briefly identifies overfitting and loss variants.
- Examples
- Uses UltraFeedback Binarized preference splits to train and save a model with DPOTrainer.
- What you can do after
- Apply it via TRL's DPOTrainer by preparing preference fields, configuring training, and saving the resulting model.
Read how Transformers in Action explains Direct Preference Optimization →
Also covered in
- Machines that ThinkIn early access (MEAP)
- LLM Evaluation and Alignment, The Foundational IdeasIn early access (MEAP)
- Introduction to Generative AI, Second Edition
- Generative AI in Action
What to know first
- fine-tuningFine-tuning is the continued training of a pretrained model on targeted data or objectives to adapt its parameters or behavior to a task, domain, or use.
- RLHFReinforcement learning from human feedback is a model-training approach that uses human evaluations or preferences as feedback to optimize model behavior.
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- reward modelA reward model is a learned model that assigns scores to outputs, actions, or outcomes to represent their desirability according to preferences or evaluation criteria.
Related concepts
- RLHFcontrasts with · Reinforcement learning from human feedback is a model-training approach that uses human evaluations or preferences as feedback to optimize model behavior.
- preference datauses · Preference data is a collection of judgments about the relative preference or quality of alternatives, often candidate outputs for the same input.
- Group Relative Policy Optimizationequivalent to · Group Relative Policy Optimization is a reinforcement-learning algorithm that updates a policy using relative rewards or advantages estimated from groups of sampled actions or outputs.
- preference pairuses · A preference pair is a pair of candidate responses to the same prompt, accompanied by an indication of which response is preferred.
- preference tuningused by · Preference tuning is a model-training technique that adjusts a model’s behavior toward outputs favored by preference signals, judgments, objectives, or rewards.
Questions about Direct Preference Optimization
What is Direct Preference Optimization?
Direct Preference Optimization is a technique for training a model directly from preference data, such as preferred and dispreferred outputs, without separately fitting a reward model.
Which Manning books explain Direct Preference Optimization?
LLM Customization and Fine-Tuning, Reinforcement Learning from Human Feedback and Transformers in Action.
What should I know before learning Direct Preference Optimization?
Manning books that teach Direct Preference Optimization usually introduce fine-tuning, RLHF, language model and reward model first.
What is the difference between Direct Preference Optimization and RLHF?
Direct Preference Optimization is a technique for training a model directly from preference data, such as preferred and dispreferred outputs, without separately fitting a reward model. Reinforcement learning from human feedback is a model-training approach that uses human evaluations or preferences as feedback to optimize model behavior.
What is Direct Preference Optimization also called?
Direct Preference Optimization is also called DPO.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


