Chapter 8: Direct-alignment algorithms
The introduction presents direct-alignment algorithms as a simpler way to optimize the RLHF preference objective and identifies DPO as a prominent example.
After this chapter you will be able to
- Direct alignment approach. Explain that direct alignment algorithms update language models using the same RLHF preference data and objective without training an intermediate reward model or using reinforcement learning optimizers.
- DPO optimization principle. State that direct preference optimization is a prominent DAA that uses gradient ascent to solve the same constrained RLHF objective.
- Reference-relative preference loss. Explain how DPO compares the policy’s probability shifts for chosen and rejected completions relative to a reference model, and when the loss decreases.
- Implicit reward model. Explain why DPO can optimize from preferences without learning a separate, explicit reward model, while still relying on an implicit one.
- KL balance and offline data. Explain β’s role in balancing preference optimization against distance from the reference model, and distinguish DPO’s fixed preference-data setting from online policy-gradient training.
- Optimal RLHF policy form. Explain how the KL-regularized RLHF objective becomes a minimization of KL divergence to a normalized, reward-weighted reference policy, and why that distribution is optimal.
- Preference likelihood to DPO loss. Trace how solving the optimal-policy expression for reward and substituting it into the Bradley–Terry preference model produces a sigmoid likelihood and the DPO negative-log-likelihood loss.
- Unequal preference pair signals. Explain why treating every preference pair equally can be a limitation of DPO and describe how variants use reward margins, label noise, or offsets to change the training signal.
- DPO preference displacement. Explain how increasing the chosen-versus-rejected probability margin can lower both responses’ probabilities, and distinguish the proposed consequence from an established outcome.
- Stale versus online completions. Explain the proposed performance limitation of training on completions from previous or other models and how online DPO variants address it.
- DPO loss computation. Trace how chosen and rejected completion log-probabilities from the policy and frozen reference models produce the DPO loss, and distinguish the logged implicit rewards from the loss used for gradients.
- Synthetic preference data as a starting point. Explain what synthetic preference data is in this setting and why the section recommends starting with it for DAA training.
- Competing response-pair construction views. Distinguish the on-policy and delta-learning views of what matters when constructing synthetic preference pairs.
- Preference margins and token gradients. Explain the high-level probability and token-gradient intuition for how most DAAs change model generations.
- RL and DAA performance. Describe the reported performance difference between policy-gradient-based RL and DPO variants, and explain why the comparison is more nuanced than asking whether RL is necessary for alignment.
- Practical value of DAAs. Explain why DAAs remain widely used despite the reported performance gap with RL.
What it builds on
- Problem formulation (chapter 3) · The RLHF problem formulation provides the objective being discussed.
- Training a Bradley–Terry reward model (chapter 5) · Preference comparisons in a Bradley–Terry model.
- Policy gradient algorithms (chapter 6) · Policy-gradient methods used for comparison.
- The role of reinforcement learning in RLHF (chapter 6) · The role of the RLHF optimization objective.
Section by section
Introduction
The introduction presents direct-alignment algorithms as a simpler way to optimize the RLHF preference objective and identifies DPO as a prominent example.
- Direct alignment approach: Explain that direct alignment algorithms update language models using the same RLHF preference data and objective without training an intermediate reward model or using reinforcement learning optimizers.
Why it matters: This distinction explains what is direct about these algorithms without treating them as a different preference-learning task. - DPO optimization principle: State that direct preference optimization is a prominent DAA that uses gradient ascent to solve the same constrained RLHF objective.
Why it matters: It connects the chapter's main example to the RLHF objective already introduced. - Practical appeal of DAAs: Explain why removing the reward-model and online-optimization steps makes DAAs attractive for post-training experiments.
Why it matters: The practical savings explain why these methods matter beyond their mathematical formulation.
Key terms: direct alignment algorithms, DAAs, reward model, preference learning, direct preference optimization, DPO, gradient ascent, constrained RLHF objective, online optimization, training compute
8.1 Direct preference optimization
8.1.1 How DPO works
This section explains how DPO uses preference pairs, an implicit reward model, and a reference-relative loss to update a policy without a separate reward-model training and sampling loop.
- Reference-relative preference loss: Explain how DPO compares the policy’s probability shifts for chosen and rejected completions relative to a reference model, and when the loss decreases.
Why it matters: This comparison is the central signal that tells DPO which response to favor. - Implicit reward model: Explain why DPO can optimize from preferences without learning a separate, explicit reward model, while still relying on an implicit one.
Why it matters: It resolves the apparent contradiction between directly training a policy and learning from a reward signal. - KL balance and offline data: Explain β’s role in balancing preference optimization against distance from the reference model, and distinguish DPO’s fixed preference-data setting from online policy-gradient training.
Why it matters: This explains both a consequential DPO tuning choice and a central distinction from online RL.
Key terms: chosen completion, rejected completion, reference model, log-ratio, DPO loss, implicit RM, explicit RM, optimal policy, β, KL distance
8.1.2 DPO derivation
The section derives DPO by finding the optimal KL-regularized RLHF policy, expressing pairwise preference likelihood in terms of that policy, and differentiating the resulting loss.
- Optimal RLHF policy form: Explain how the KL-regularized RLHF objective becomes a minimization of KL divergence to a normalized, reward-weighted reference policy, and why that distribution is optimal.
Why it matters: This optimal-policy form is the link that lets DPO replace explicit reward optimization with a preference-based loss. - Preference likelihood to DPO loss: Trace how solving the optimal-policy expression for reward and substituting it into the Bradley–Terry preference model produces a sigmoid likelihood and the DPO negative-log-likelihood loss.
Why it matters: This connection explains why DPO can train directly on preferred-versus-rejected responses. - DPO loss gradient derivation: Describe how the DPO gradient is obtained by differentiating the negative log-sigmoid loss with respect to model parameters, using the preferred and rejected responses’ log policy-to-reference ratios.
Why it matters: It connects the derived DPO loss to the parameter update discussed earlier in the chapter.
Key terms: partition function, reference policy, KL divergence, optimal policy, Gibbs’ inequality, Bradley–Terry model, DPO loss, negative log-likelihood, sigmoid, DPO gradient
8.2 Numerical concerns, weaknesses, and alternatives
This section explains DPO’s main weaknesses, how several direct-alignment variants address them, and why data and the initial model may matter more than the algorithm choice.
- Unequal preference pair signals: Explain why treating every preference pair equally can be a limitation of DPO and describe how variants use reward margins, label noise, or offsets to change the training signal.
Why it matters: It explains why changing the information carried by a preference pair can change DPO training. - Loss and setup changes: Describe how ORPO and SimPO change DPO-inspired training, distinguishing ORPO’s reference-model-free setup from SimPO’s length normalization.
Why it matters: These examples show that direct-alignment alternatives can change either the training setup or how response likelihoods enter the loss. - DPO preference displacement: Explain how increasing the chosen-versus-rejected probability margin can lower both responses’ probabilities, and distinguish the proposed consequence from an established outcome.
Why it matters: A wider preference margin does not necessarily mean the chosen response itself has become more likely. - Stale versus online completions: Explain the proposed performance limitation of training on completions from previous or other models and how online DPO variants address it.
Why it matters: It connects the source and freshness of training data to a proposed reason online methods can outperform offline DPO. - Algorithm choice relative importance: State how the section ranks the choice among DAA variants against the initial model and training data.
Why it matters: It puts the long list of algorithm variants in perspective when considering what most affects training.
Key terms: DPO, preference data, REBEL, cDPO, IPO, ODPO, ORPO, SimPO, reference model, log-probabilities
8.3 Implementation details
This section shows how DPO’s loss is assembled in a language-model training stack and explains two implementation considerations: its fixed KL tradeoff and reference-model memory use.
- DPO loss computation: Trace how chosen and rejected completion log-probabilities from the policy and frozen reference models produce the DPO loss, and distinguish the logged implicit rewards from the loss used for gradients.
Why it matters: This explains why DPO can use a standard language-model training stack with an added reference model. - Fixed KL tradeoff: Explain how the beta parameter sets DPO’s distance penalty and contrast DPO’s steps toward the data-dependent RLHF solution with RL-based optimizers’ steps based on batches and recent data.
Why it matters: The comparison clarifies that DPO and RL-based optimization use different kinds of training steps. - Reference log-probability caching: Describe why computing policy and reference forward passes together raises memory use and how precomputing reference log-probabilities for the dataset reduces peak memory during training.
Why it matters: Caching makes the same loss calculation less demanding on GPU memory.
Key terms: DPO loss, policy model, frozen reference model, log-ratios, negative log-sigmoid, implicit rewards, KL divergence, beta, distance penalty, RL-based optimizers
8.4 DAAs with synthetic preference data
The section explains why synthetic preference data is a practical starting point for DAAs, how response pairs and judge biases affect that data, and how DAA training changes completion probabilities.
- Synthetic preference data as a starting point: Explain what synthetic preference data is in this setting and why the section recommends starting with it for DAA training.
Why it matters: This explains why synthetic comparisons are a common practical choice for preference fine-tuning. - Competing response-pair construction views: Distinguish the on-policy and delta-learning views of what matters when constructing synthetic preference pairs.
Why it matters: The two views focus attention on different choices when assembling preference pairs. - Synthetic judge bias: Describe how biases in a model judging completions can affect which responses appear as chosen.
Why it matters: A synthetic preference label can reflect the judge’s tastes as well as a comparison between responses. - Preference margins and token gradients: Explain the high-level probability and token-gradient intuition for how most DAAs change model generations.
Why it matters: This prevents equating preference training with simply raising the probability of every chosen response.
Key terms: synthetic preferences, DAAs, preference fine-tuning, on-policy, contrastive, delta learning, chosen, rejected, length bias, model judging
8.5 DAAs vs. RL: Online vs. offline data
The section weighs RL’s reported performance advantage over direct-alignment algorithms against the practical reasons DAAs remain widely used.
- RL and DAA performance: Describe the reported performance difference between policy-gradient-based RL and DPO variants, and explain why the comparison is more nuanced than asking whether RL is necessary for alignment.
Why it matters: The performance comparison helps explain why choosing between these methods is not a simple yes-or-no decision about RL. - Practical value of DAAs: Explain why DAAs remain widely used despite the reported performance gap with RL.
Why it matters: The best reported performance is not the only consideration when selecting a training approach.
Key terms: policy-gradient-based, RL methods, DPO, on-policy data, DAAs, training data, configurations
8.6 Summary
Where this is used later
- AI feedback (chapter 12) · Connects to the later treatment of AI feedback data.
Concepts explained in this chapter
- Direct Preference OptimizationDirect preference optimization (DPO) directly trains a policy from pairwise preferred and dispreferred outputs relative to a reference policy.
- Kullback–Leibler divergenceForward KL divergence is KL divergence from a target distribution to an approximating distribution, with expectation taken under the target distribution.
- DPO lossThe DPO loss is a differentiable preference-training objective that increases the relative likelihood of chosen completions over rejected completions relative to a reference policy.
- implicit rewardAn implicit reward is a reward value inferred from a preference-optimization objective rather than predicted by a separately trained reward model.
- log-ratioA log-ratio is the logarithm of the ratio between two positive quantities, such as model and reference probabilities.
- direct alignment algorithmA direct alignment algorithm optimizes a model directly from preference data without learning an intermediate reward model.
- Bradley-Terry modelA Bradley–Terry model is a probabilistic model that represents pairwise preferences using latent positive strengths or equivalent real-valued scores for items.
- negative log-likelihoodNegative log-likelihood is the negation of a model's log-likelihood, used as a loss for probabilistic predictions.
- partition functionA partition function is a normalization factor that converts unnormalized nonnegative weights into a probability distribution.
- synthetic preference dataSynthetic preference data is preference data generated by models or algorithmic processes rather than directly collected from human judgment.
- gradientA gradient is a vector of derivatives indicating how a scalar function changes with respect to its parameters.
- preference displacementPreference displacement is a shift in probability mass away from responses represented in preference-training data toward unaddressed model behaviors.
- preference marginA preference margin is the numerical difference in reward or rating values between two compared outputs, representing the strength of preference.
- online DPOOnline DPO is a DPO variant that samples new completions from the current model during training and incorporates preference signals at training time.
- contrastive loss functionA contrastive loss is an optimization objective that compares examples to encourage similar or preferred examples to be distinguished from dissimilar or non-preferred ones.
- length biasLength bias is a systematic preference for outputs of particular lengths that can distort model optimization or evaluation.
- on-policy methodOn-policy optimization, or on-policy learning, updates a policy using data generated by that same policy or its current update distribution.
- preference dataA pairwise comparison is an evaluation that determines which of two items is preferred.
- objective functionA loss function is a scalar objective that quantifies prediction or optimization error during model training.
- reward signalA reward is a scalar evaluative signal indicating the desirability of an action or outcome for an agent.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.