chapter eight

8 Direct-alignment algorithms

 

This chapter covers

  • The derivation of the first DAA, direct preference optimization (DPO), from first principles
  • Intuitions for DPO and other related algorithms
  • What to consider when using DAAs

Direct alignment algorithms (DAAs) let us update models to solve the same RLHF objective without ever training an intermediate reward model (RM) or using reinforcement learning optimizers. DAAs solve the same preference learning problem we’ve been studying (with literally the same data!) to make language models more aligned, smarter, and easier to use. The lack of an RM and online optimization makes DAAs far simpler to implement, reducing compute spent during training and making experimentation easier. This chapter details the complex mathematics done to derive these algorithms and then shows that the sometimes tedious derivations result in simple implementations.

The most prominent DAA, and one that catalyzed an entire academic movement of aligning language models, is direct preference optimization (DPO) [1]. At its core, DPO uses gradient ascent to solve the same constrained RLHF objective (see chapter 3):

Equation 8.1
\[\label{eq:review_rlhf} \max_{\pi} \mathbb{E}_{x \sim \mathcal{D}}\mathbb{E}_{y \sim \pi(y|x)} \left[r_\theta(x, y)\right] - \beta \mathcal{D}_{\text{KL}}\left(\pi(y|x) \| \pi_{\text{ref}}(y|x)\right).\]

8.1 Direct preference optimization

8.1.1 How DPO works

8.1.2 DPO derivation

8.2 Numerical concerns, weaknesses, and alternatives

8.3 Implementation details

8.4 DAAs with synthetic preference data

8.5 DAAs vs. RL: Online vs. offline data

Summary