Direct Preference Optimization
Direct Preference Optimization is a technique for training a model directly from preference data, such as preferred and dispreferred outputs, without separately fitting a reward model.
Reinforcement Learning from Human Feedback
Direct preference optimization (DPO) directly trains a policy from pairwise preferred and dispreferred outputs relative to a reference policy.
- How it is used
- Fine-tunes language models from offline preference pairs while bypassing a reward model and RL optimization.
- How deep it goes
- Derives the closed-form loss from the KL-regularized RLHF objective and Bradley–Terry model, then details forward-pass implementation and memory caching.
- Examples
- Uses chosen and rejected prompt completions, policy and reference log-probabilities, and a code-like loss calculation.
- What you can do after
- Apply it via standard language-model training stacks using paired completions, a frozen reference model, and the DPO loss.
Read how Reinforcement Learning from Human Feedback explains Direct Preference Optimization →
Transformers in Action
Direct Preference Optimization is a preference-alignment technique that directly optimizes a language model from chosen and rejected responses without separately training an explicit reward model.
- How it is used
- Aligns language-model outputs with human preferences by optimizing chosen responses over rejected responses without a separate reward model.
- How deep it goes
- Explains preference pairs and KL divergence, then gives dataset preparation and trainer steps; stops short of deriving the loss.
- Examples
- Uses UltraFeedback Binarized preference splits, chat-template transformation, DPOTrainer configuration, and local or Hub model saving.
- What you can do after
- Apply it via TRL’s DPOTrainer to fine-tune a previously trained model on formatted preference data.
Read how Transformers in Action explains Direct Preference Optimization →
Related concepts
- reinforcement learning with human feedbackcontrasts with · Reinforcement learning with human feedback is a model-training approach that uses human evaluations or preferences to guide optimization of model behavior.
- preference datauses · Preference data is a collection of judgments or comparisons indicating the relative preference or quality of alternatives, often model outputs.
- Group Relative Policy Optimizationequivalent to · Group Relative Policy Optimization is a reinforcement-learning algorithm that updates a policy using relative rewards or advantages estimated from groups of sampled outputs or actions.
- preference tuningused by · Preference tuning is a model-training technique that adjusts a model’s behavior toward outputs favored by preference signals, judgments, objectives, or rewards.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.