Direct Preference Optimization (DPO) trains a language-model policy from pairs of preferred and dispreferred completions. It targets the KL-regularized RLHF objective without training a separate reward model or using a reinforcement learning optimizer. Unlike policy-gradient methods, DPO is off-policy by default. section 6.2.11 section 8.0
For a prompt x, let y_c be the chosen completion, y_r the rejected completion, πθ the trainable policy, and πref the frozen reference policy. The loss compares each completion’s log-probability shift from the reference. It decreases when the chosen shift exceeds the rejected shift. Sequence log-probabilities are sums of token log-probabilities. section 8.1.1 section 8.3
Equation 8.2
\[\label{eq:dpo_core} \mathcal{L}_{\text{DPO}}(\pi_\theta; \pi_{\text{ref}}) = -\mathbb{E}_{(x, y_c, y_r) \sim \mathcal{D}}\left[ \log \sigma\left( \beta \log \frac{\pi_{\theta}(y_c \mid x)}{\pi_{\text{ref}}(y_c \mid x)} - \beta \log \frac{\pi_{\theta}(y_r \mid x)}{\pi_{\text{ref}}(y_r \mid x)} \right) \right].\]The negative log-sigmoid turns the difference between chosen and rejected log-ratios into a loss. β balances reward optimization against distance from the reference. section 8.1.1
The derivation starts with the optimal policy for a reward objective penalized by KL divergence from the reference. Expressing its reward in terms of the policy-to-reference log-ratio and substituting it into a Bradley–Terry preference model eliminates the prompt-specific normalization term. The resulting pairwise likelihood can be optimized directly, without fitting an explicit reward model. section 8.1.2
pi_logratios = policy_chosen_logps - policy_rejected_logps 1
ref_logratios = reference_chosen_logps - reference_rejected_logps2
logits = pi_logratios - ref_logratios3
losses = -F.logsigmoid(beta * logits)4
chosen_rewards = (
beta * (policy_chosen_logps - reference_chosen_logps).detach()
)5
rejected_rewards = (
beta * (policy_rejected_logps - reference_rejected_logps).detach()
)
Compute policy and frozen-reference log-probabilities for both completions, subtract their chosen–rejected differences, and apply the negative log-sigmoid. Detached implicit rewards can be logged separately. section 8.3
Caching reference log-probabilities over the training dataset avoids running both models together for each batch and can reduce peak memory usage. Learning rate and β require care: successful early DPO models used surprisingly low learning rates. section 8.3 section 8.0
A larger relative preference margin does not guarantee that the chosen completion becomes more probable. In preference displacement, probabilities of both completions fall, but the rejected one falls further. Offline data and this displacement are posited reasons for a peak-performance gap with RL-based methods. Variants address other limitations: cDPO and IPO target label noise, while SimPO adds length normalization. section 8.6 section 8.2


