A direct alignment algorithm (DAA) fine-tunes a language-model policy from pairwise preference data: given a prompt, a chosen completion, and a rejected completion, it improves the policy without first fitting a separate reward model. It targets the same preference-learning objective as RLHF policy-gradient methods, while avoiding both an intermediate reward model and reinforcement-learning optimization. section 3.1.5 section 8.0
The practical motivation is to make preference fine-tuning cheaper and easier to iterate on. Traditional RLHF can instruction-tune a model, train a reward model on preference pairs, then optimize the policy against that learned reward. A DAA instead uses the preference pairs directly. This removes a training component and online optimization loop, reducing compute and lowering the implementation barrier for toy and production experiments. section 8.0 section 3.3
Equation 8.1
\[\label{eq:review_rlhf} \max_{\pi} \mathbb{E}_{x \sim \mathcal{D}}\mathbb{E}_{y \sim \pi(y|x)} \left[r_\theta(x, y)\right] - \beta \mathcal{D}_{\text{KL}}\left(\pi(y|x) \| \pi_{\text{ref}}(y|x)\right).\]The shared constrained RLHF objective rewards high-quality responses while penalizing divergence from a reference policy. DAAs solve this preference problem through a direct loss rather than separately learning the reward term and applying a policy-gradient optimizer. section 8.0
Direct preference optimization (DPO) is the prominent DAA. Its closed-form loss follows from the Bradley–Terry preference model: the optimal policy implicitly defines a reward function. Operationally, DPO increases the margin between chosen and rejected completions according to how incorrect that implicit reward currently is. The result is a gradient-based implementation built around relative likelihoods of the two responses, rather than sampled online rollouts scored by a reward model. section 8.6 section 6.2.11
Synthetic preference pipeline
A common pipeline uses a frontier model to judge outputs from other models and label a winner and loser for each prompt. Training then fine-tunes the target model on those synthetic pairs. For best results, some completions may be generated by the model being fine-tuned, mixed with a broader model pool; another supported view emphasizes the quality of the difference between chosen and rejected completions. Judge-model biases remain a risk, including preferences for longer outputs or stylistically similar responses. section 8.4
Key trade-offs and variants include:
- DAAs are off-policy by default, unlike policy-gradient methods derived as on-policy algorithms. section 6.2.11
- DPO can cause preference displacement: probabilities of both chosen and rejected responses fall, though rejected responses fall more; offline data may contribute to RL methods' slightly higher peak performance. section 8.6
- IPO and cDPO address label noise; SimPO adds length normalization; ORPO removes the reference model; online DPO refreshes completions during training. section 8.6
- DAAs can still over-optimize learned preference signals, but fixed KL distances offer more direct control of the optimization trade-off. section 14.2 section 14.4
DAAs are widely used for preference fine-tuning because they preserve the core alignment goal while simplifying the pipeline. Synthetic preference data is a practical starting point, but data provenance, chosen-versus-rejected contrast, model-judge bias, and KL-controlled over-optimization remain central design concerns. section 8.4 section 8.6
