liveBook · Manning

Direct Preference Optimization

As explained in Reinforcement Learning from Human Feedback

Direct preference optimization (DPO) directly trains a policy from pairwise preferred and dispreferred outputs relative to a reference policy.

Direct Preference Optimization (DPO) trains a language-model policy from pairs of preferred and dispreferred completions. It targets the KL-regularized RLHF objective without training a separate reward model or using a reinforcement learning optimizer. Unlike policy-gradient methods, DPO is off-policy by default. section 6.2.11 section 8.0

For a prompt x, let y_c be the chosen completion, y_r the rejected completion, πθ the trainable policy, and πref the frozen reference policy. The loss compares each completion’s log-probability shift from the reference. It decreases when the chosen shift exceeds the rejected shift. Sequence log-probabilities are sums of token log-probabilities. section 8.1.1 section 8.3

Equation 8.2
\[\label{eq:dpo_core} \mathcal{L}_{\text{DPO}}(\pi_\theta; \pi_{\text{ref}}) = -\mathbb{E}_{(x, y_c, y_r) \sim \mathcal{D}}\left[ \log \sigma\left( \beta \log \frac{\pi_{\theta}(y_c \mid x)}{\pi_{\text{ref}}(y_c \mid x)} - \beta \log \frac{\pi_{\theta}(y_r \mid x)}{\pi_{\text{ref}}(y_r \mid x)} \right) \right].\]

The negative log-sigmoid turns the difference between chosen and rejected log-ratios into a loss. β balances reward optimization against distance from the reference. section 8.1.1

The derivation starts with the optimal policy for a reward objective penalized by KL divergence from the reference. Expressing its reward in terms of the policy-to-reference log-ratio and substituting it into a Bradley–Terry preference model eliminates the prompt-specific normalization term. The resulting pairwise likelihood can be optimized directly, without fitting an explicit reward model. section 8.1.2

pi_logratios = policy_chosen_logps - policy_rejected_logps   1
ref_logratios = reference_chosen_logps - reference_rejected_logps2

logits = pi_logratios - ref_logratios3

losses = -F.logsigmoid(beta * logits)4

chosen_rewards = (
    beta * (policy_chosen_logps - reference_chosen_logps).detach()
)5

rejected_rewards = (
    beta * (policy_rejected_logps - reference_rejected_logps).detach()
)

Compute policy and frozen-reference log-probabilities for both completions, subtract their chosen–rejected differences, and apply the negative log-sigmoid. Detached implicit rewards can be logged separately. section 8.3

Caching reference log-probabilities over the training dataset avoids running both models together for each batch and can reduce peak memory usage. Learning rate and β require care: successful early DPO models used surprisingly low learning rates. section 8.3 section 8.0

A larger relative preference margin does not guarantee that the chosen completion becomes more probable. In preference displacement, probabilities of both completions fall, but the rejected one falls further. Offline data and this displacement are posited reasons for a peak-performance gap with RL-based methods. Variants address other limitations: cDPO and IPO target label noise, while SimPO adds length normalization. section 8.6 section 8.2

How Reinforcement Learning from Human Feedback covers it

How deep it goes
Derives the objective from the optimal KL-regularized policy and Bradley–Terry preferences, then details the loss, gradient, implementation, and weaknesses.
Examples
Uses a loss implementation and cites deployed models including Zephyr-Beta and Tülu 2.
What it assumes
Assumes familiarity with KL divergence and the Bradley–Terry preference model while deriving their application to DPO.
What you can do after
Builds DPO from scratch using policy and reference log-probabilities, with implicit rewards for logging and optional reference-probability caching.
How it connects
Connects RLHF, KL divergence, Bradley–Terry preference modeling, and policy-gradient methods.

Also explained in

Compare how 3 books explain Direct Preference Optimization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.