8 Preference optimization: Teaching your model to judge
This chapter covers
- Training Direct Preference Optimization (DPO) on preference pairs using the Transformer Reinforcement Learning (TRL) library’s DPOTrainer
- Evaluating base, SFT, and DPO models with the progressive evaluation pipeline
- Why SFT plateaus on style, helpfulness, and safety boundary tuning
- Comparing Group Relative Policy Optimization (GRPO), Reinforcement Learning from Human Feedback (RLHF), and Reinforcement Fine-Tuning (RFT) tradeoffs
- Testing safety regression after preference tuning
In chapter 6, we saw that the SFT model improved quality substantially across the target domain. On the IT help-desk evaluation, Token-F1 moved only a little (from 0.153 base to 0.168 SFT on the held-out test split). Still, the model learned to follow your system prompt, produce concise answers, and match the formatting conventions in your training data (format adherence jumped from 0% to 100%). But some quality dimensions resist supervised learning entirely. Two responses can both be factually correct, and yet one is clearly better: more helpful, better structured, more appropriately detailed. SFT cannot teach this distinction because it only sees positive examples. Preference optimization addresses this gap by training the model on response pairs (one better, one worse), so it learns to judge rather than merely imitate.