Direct preference optimization (DPO) is a preference-fine-tuning method that reaches the RLHF objective without training a separate reward model or running reinforcement-learning optimization. It trains directly on offline pairs: a prompt, a chosen completion, and a rejected completion. This reduces the moving parts, compute, and experimentation cost of alignment training. section 8.0 section 6.2.11
For each pair, DPO asks whether the policy has increased the chosen completion’s probability more than the rejected completion’s probability, relative to a frozen reference policy. The negative log-sigmoid loss falls as that relative gap grows. β controls the trade-off between preference optimization and staying close to the reference policy; it is therefore crucial for limiting over-optimization. section 8.1.1 section 8.3
Equation 8.2
\[\label{eq:dpo_core} \mathcal{L}_{\text{DPO}}(\pi_\theta; \pi_{\text{ref}}) = -\mathbb{E}_{(x, y_c, y_r) \sim \mathcal{D}}\left[ \log \sigma\left( \beta \log \frac{\pi_{\theta}(y_c \mid x)}{\pi_{\text{ref}}(y_c \mid x)} - \beta \log \frac{\pi_{\theta}(y_r \mid x)}{\pi_{\text{ref}}(y_r \mid x)} \right) \right].\]The core loss compares policy-to-reference log-probability shifts for chosen and rejected completions, then converts their difference into a pairwise preference objective. section 8.1.1
A standard training step needs only forward-pass log-probabilities for both completions:
- Run the trainable policy on the chosen and rejected completions, and subtract their sequence log-probabilities. section 8.3
- Run the frozen reference policy on the same pairs, then subtract its chosen and rejected log-probabilities. section 8.3
- Subtract the reference ratio from the policy ratio, scale by β, and apply negative log-sigmoid as the loss. section 8.3
- Optionally precompute and cache reference log-probabilities to avoid concurrent reference forward passes, reducing peak memory use by 50%. section 8.3
DPO follows from the KL-regularized RLHF objective and a Bradley–Terry preference model: the optimal policy defines an implicit reward through its log-ratio against the reference policy. This lets pairwise preference likelihood replace an explicitly learned reward model. section 8.1.2 section 8.1.1
Its simplicity does not make it universally optimal. Offline binary pairs receive equal weight, label noise can cause overfitting, and preference displacement may reduce probabilities for both completions while reducing the rejected one more. IPO and cDPO target noisy labels, SimPO adds length normalization, ORPO removes the reference model, and online DPO refreshes completions during training. section 8.2 section 8.6
Preference-pair update
Given one prompt with a preferred answer and a dispreferred answer, DPO increases the preferred answer’s probability relative to its reference probability more than it increases—or decreases less than—the dispreferred answer’s corresponding probability. The update is computed from whole-completion token log-probability sums, not from a separately scored rollout. section 8.1.1 section 8.3
