Negative log-likelihood (NLL) turns a modeled probability into a minimization objective: minimizing −log p is equivalent to maximizing p. In preference learning, p is the probability that a chosen completion is preferred over a rejected completion, rather than the probability of a single class or token. section 5.1 section 8.1.2
For a Bradley–Terry reward model, let Δ = rθ(yc | x) − rθ(yr | x). The pairwise NLL is −log σ(Δ), equivalently log(1 + e^(rθ(yr | x) − rθ(yc | x))). Reducing it increases the chosen completion’s score relative to the rejected one. section 5.1
DPO as preference NLL
DPO applies NLL to a Bradley–Terry preference probability built from policy-to-reference log-likelihood ratios. Its sigmoid argument compares how much the trainable policy raises the chosen completion’s likelihood relative to the rejected completion, with β controlling that comparison. section 8.1.2
NLL can also regularize preference optimization. Adding a length-normalized NLL penalty on winning completions keeps preferred text high-likelihood in an absolute language-modeling sense, rather than merely more likely than its rejected alternative. section 15.3.1 section 15.4
On supervised fine-tuning data, dataset NLL is the standard cross-entropy loss. Up to a constant negative-entropy term, minimizing that loss has the same gradients and minimum as minimizing forward KL from the data distribution to the policy. section 15.2.2
Equation 8.28
\[\label{eq:dpo_loss_deriv4} \begin{aligned}[b] \mathcal{L}_{\text{DPO}}(\pi_{\theta}; \pi_{\text{ref}}) &= -\mathbb{E}_{(x,y_c,y_r)\sim\mathcal{D}}\left[ \log p(y_c \succ y_r \mid x) \right] \\ &= -\mathbb{E}_{(x,y_c,y_r)\sim\mathcal{D}}\left[ \log \sigma\left(\beta \log \frac{\pi_{\theta}(y_c|x)}{\pi_{\text{ref}}(y_c|x)} - \beta \log \frac{\pi_{\theta}(y_r|x)}{\pi_{\text{ref}}(y_r|x)}\right)\right] \end{aligned}.\]Inspect the full DPO objective: it makes the expectation of negative log preference probability explicit and shows the chosen/rejected likelihood-ratio comparison. section 8.1.2
