liveBook · Manning

Direct Preference Optimization

Also called DPO

Direct Preference Optimization is a technique for training a model directly from preference data, such as preferred and dispreferred outputs, without separately fitting a reward model.

How 3 Manning books explain it

LLM Customization and Fine-Tuning

A method that tunes a model directly from preference comparisons without separately training a reward model.

How it is used
Uses DPO after supervised fine-tuning to refine subjective response qualities such as conciseness and organizational style.
How deep it goes
Explains the policy-reference objective and walks through preference preparation, training, evaluation, and safety testing.
Examples
Uses IT-support preference pairs and a Windows network-reset example to demonstrate concise versus verbose responses.
What you can do after
Apply DPO via TRL’s DPOTrainer and evaluate preference gains, correctness, and safety against the starting model.

Read how LLM Customization and Fine-Tuning explains Direct Preference Optimization →

Reinforcement Learning from Human Feedback

Direct preference optimization (DPO) directly trains a policy from pairwise preferred and dispreferred outputs relative to a reference policy.

How it is used
Uses pairwise preferences to align language models without a separate reward model or reinforcement learning optimizer.
How deep it goes
Derives the objective from the optimal KL-regularized policy and Bradley–Terry preferences, then details the loss, gradient, implementation, and weaknesses.
Examples
Uses a loss implementation and cites deployed models including Zephyr-Beta and Tülu 2.
What you can do after
Builds DPO from scratch using policy and reference log-probabilities, with implicit rewards for logging and optional reference-probability caching.

Read how Reinforcement Learning from Human Feedback explains Direct Preference Optimization →

Transformers in Action

Direct Preference Optimization is a preference-alignment technique that directly optimizes a language model from chosen and rejected responses without separately training an explicit reward model.

How it is used
Aligns language-model responses with human preferences by optimizing chosen over rejected outputs.
How deep it goes
Explains the preference-training workflow and trainer configuration, then briefly identifies overfitting and loss variants.
Examples
Uses UltraFeedback Binarized preference splits to train and save a model with DPOTrainer.
What you can do after
Apply it via TRL's DPOTrainer by preparing preference fields, configuring training, and saving the resulting model.

Read how Transformers in Action explains Direct Preference Optimization →

Also covered in

What to know first

Related concepts

Questions about Direct Preference Optimization

What is Direct Preference Optimization?

Direct Preference Optimization is a technique for training a model directly from preference data, such as preferred and dispreferred outputs, without separately fitting a reward model.

Which Manning books explain Direct Preference Optimization?

LLM Customization and Fine-Tuning, Reinforcement Learning from Human Feedback and Transformers in Action.

What should I know before learning Direct Preference Optimization?

Manning books that teach Direct Preference Optimization usually introduce fine-tuning, RLHF, language model and reward model first.

What is the difference between Direct Preference Optimization and RLHF?

Direct Preference Optimization is a technique for training a model directly from preference data, such as preferred and dispreferred outputs, without separately fitting a reward model. Reinforcement learning from human feedback is a model-training approach that uses human evaluations or preferences as feedback to optimize model behavior.

What is Direct Preference Optimization also called?

Direct Preference Optimization is also called DPO.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.