In DPO, KL regularization constrains how far the trainable policy moves from a frozen copy of the pre-DPO model. This lets training favor chosen responses and disfavor rejected ones while limiting changes to the model’s existing behavior. section 8.2.5
In TRL’s DPO setup, beta controls the strength of this constraint:
- Higher beta keeps the policy closer to its reference; lower beta permits more aggressive preference learning. section 8.2.5
- The described configuration uses beta 0.1 as a baseline. Below 0.05, the policy can drift or collapse toward a narrow set of outputs; above 0.3, it may barely change. section 8.2.3
A conservative starting point
For a model already refined with supervised fine-tuning, start DPO at beta 0.1 and establish a baseline before adjusting. The example run uses 270 preference pairs, one epoch, and a learning rate of 5e-6; beta helps limit drift while the policy learns from those pairs. section 8.2.3
Figure 8.3 The DPO training mechanism: raise the chosen response, lower the rejected one, stay near the frozen reference.

The diagram shows the policy raising the chosen response, lowering the rejected response, and staying near the frozen reference. section 8.2.5




