Preference training can improve a model’s chosen responses while disturbing capabilities it already has. A reference model provides a fixed comparison point: in direct preference optimization (DPO), it is a frozen copy of the pre-DPO checkpoint, while a separate policy model is updated. section 8.2.5 section 8.2.3
DPO uses chosen and rejected responses to shift the policy’s preferences relative to the reference. Its implicit reward compares the policy’s response probability with the frozen reference’s probability, scaled by beta. The reference anchors the update without requiring a separate reward model or reinforcement learning loop. section 8.2 section 8.2.5
Figure 8.3 The DPO training mechanism: raise the chosen response, lower the rejected one, stay near the frozen reference.

The diagram shows how chosen and rejected responses drive policy updates while the frozen reference constrains drift. section 8.2.5
Set the reference constraint deliberately:
- Start with a strong supervised fine-tuning (SFT) checkpoint. A weak checkpoint becomes a weak reference, and DPO inherits its weaknesses. section 8.8.3
- Use beta 0.1 as a baseline in TRL. Higher beta keeps updates closer to the reference; below 0.05 allows more aggressive changes but risks destabilizing other capabilities. Above 0.3, the policy may barely change. section 8.2.5 section 8.2.3
- Budget for both model copies and their forward passes. If memory is constrained, consider reference-free ORPO or SimPO rather than assuming standard DPO is the only option. section 8.2.11
Memory planning for a 4B model
One full-parameter run needed about 54 GB for DPO, compared with about 32 GB for full SFT, because policy and reference copies were loaded together. Gradient checkpointing trades extra compute for lower memory, but this DPO run still did not fit on one 24 GB card. section 8.2.11 section 8.2.3
A reference policy also appears in reinforcement learning from human feedback (RLHF), where PPO optimizes a language model against a separately trained reward model while staying close to the reference. DPO instead uses preference pairs directly and does not train that separate reward model. section 8.4 section 8.2



