liveBook · Manning

KL regularization

As explained in LLM Customization and Fine-Tuning

A constraint based on Kullback–Leibler divergence that limits how far a policy distribution can move from a reference distribution.

In DPO, KL regularization constrains how far the trainable policy moves from a frozen copy of the pre-DPO model. This lets training favor chosen responses and disfavor rejected ones while limiting changes to the model’s existing behavior. section 8.2.5

In TRL’s DPO setup, beta controls the strength of this constraint:

  • Higher beta keeps the policy closer to its reference; lower beta permits more aggressive preference learning. section 8.2.5
  • The described configuration uses beta 0.1 as a baseline. Below 0.05, the policy can drift or collapse toward a narrow set of outputs; above 0.3, it may barely change. section 8.2.3

A conservative starting point

For a model already refined with supervised fine-tuning, start DPO at beta 0.1 and establish a baseline before adjusting. The example run uses 270 preference pairs, one epoch, and a learning rate of 5e-6; beta helps limit drift while the policy learns from those pairs. section 8.2.3

Figure 8.3 The DPO training mechanism: raise the chosen response, lower the rejected one, stay near the frozen reference.

The diagram shows the policy raising the chosen response, lowering the rejected response, and staying near the frozen reference. section 8.2.5

How LLM Customization and Fine-Tuning covers it

How deep it goes
Explains the reference-model role and beta effects, without deriving the KL divergence mathematically.
Examples
Uses a TRL beta baseline of 0.1 and describes risks below 0.05 and above 0.3.
What it assumes
Assumes an SFT-capable model, chosen and rejected response pairs, and a frozen pre-DPO reference.
What you can do after
Configure DPO’s KL strength through beta in TRL’s DPOTrainer.
How it connects
Links the constraint to DPO, reference policies, preference optimization, and RLHF with PPO.

Also explained in

Compare how 5 books explain KL regularization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.