How deeply does Reinforcement Learning from Human Feedback cover DPO loss?

Derives the objective, interprets its terms, gives implementation steps, and discusses memory caching and gradient behavior.

DPO loss

As explained in Reinforcement Learning from Human Feedback

The DPO loss is a differentiable preference-training objective that increases the relative likelihood of chosen completions over rejected completions relative to a reference policy.

The DPO loss trains a language-model policy directly from preference pairs. It increases the chosen completion’s relative likelihood over the rejected completion while constraining movement from a frozen reference policy through β, avoiding an explicit reward-model training stage. section 8.1.1

Equation 8.2

\[\label{eq:dpo_core} \mathcal{L}_{\text{DPO}}(\pi_\theta; \pi_{\text{ref}}) = -\mathbb{E}_{(x, y_c, y_r) \sim \mathcal{D}}\left[ \log \sigma\left( \beta \log \frac{\pi_{\theta}(y_c \mid x)}{\pi_{\text{ref}}(y_c \mid x)} - \beta \log \frac{\pi_{\theta}(y_r \mid x)}{\pi_{\text{ref}}(y_r \mid x)} \right) \right].\]

The objective applies a negative log-sigmoid to the difference between chosen and rejected log-likelihood shifts relative to the reference model. The loss falls when the chosen shift exceeds the rejected shift. section 8.1.1

Operationally, calculate the loss as a sequence-level preference comparison:

Run policy and reference forward passes on the chosen and rejected completions to obtain sequence log-probabilities. section 8.3
Subtract rejected log-probabilities from chosen log-probabilities for each model, then subtract the reference ratio from the policy ratio. section 8.3
Apply negative log-sigmoid to β times that difference; β controls the preference pull and effective KL constraint. section 8.1.1 section 8.3
Detach implicit chosen and rejected rewards for logging, and cache reference log-probabilities when reducing peak memory matters. section 8.3

Preference-pair behavior

For one prompt, compare a winning completion with a losing completion. If policy training raises the winner’s log-ratio over the reference more than the loser’s, the margin grows and the DPO loss decreases. A DPO+NLL variant additionally penalizes low absolute likelihood on the winning completion. section 8.1.1 section 15.3.1

How Reinforcement Learning from Human Feedback covers it

Cover of Reinforcement Learning from Human Feedback

liveBook manning.com

Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Related concepts

KL regularizationhas part · KL regularization is the use of a penalty or constraint based on Kullback–Leibler divergence to limit the difference between a learned distribution and a reference distribution.
log-ratiouses · A log-ratio is the logarithm of the ratio between two positive quantities.
negative log-likelihooduses · Negative log-likelihood is the negative logarithm of the likelihood a probabilistic model assigns to observed data, commonly used as an objective or loss to minimize.
Bradley-Terry modeluses · The Bradley-Terry model is a probabilistic model that represents pairwise preferences using latent item-strength parameters to determine preference probabilities.
reference modeluses · A reference model is a fixed policy distribution that serves as a baseline for comparing or constraining changes to an optimized policy.

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.