liveBook · Manning

KL regularization

As explained in Applied Reinforcement Learning

A regularization term based on Kullback–Leibler divergence that discourages a policy from moving too far from a reference policy.

KL regularization discourages a trainable policy from drifting too far from a reference policy. In PPO-based RLHF, the reference is usually a frozen supervised fine-tuned (SFT) model. The penalty lets training improve reward while anchoring behavior to that starting point. section 10.7

rtotal = rφ + shaping − β KL(πθ || πref)

The KL term is subtracted from reward and weighted by β; increasing its weight makes divergence more costly. section 10.7

The KL penalty and PPO clipping stabilize training in different ways: clipping limits each update relative to the policy that generated the current data, while KL regularization limits longer-term drift from the frozen reference. Those policies are not the same. section 10.7

Anchoring a medical support chatbot

In medical-chatbot RLHF, PPO fine-tuning adds a KL penalty to the reward. The frozen SFT model acts as the reference, anchoring the policy as it learns from preference rewards and safety-related shaping. section 10.7 section 10.8

The anchor’s strength depends on the model and training behavior. A math-specialized model improved without an anchor while its KL rose slowly; for instruction-tuned models, a modest β of 0.04 kept KL in a productive range and supported steady improvement. In another PPO run, an adaptive coefficient rose from 0.1 to 10 to restrain divergence from the SFT reference. section 11.6.3 section 10.8.6

How Applied Reinforcement Learning covers it

How deep it goes
Explains the reward-penalty role, distinguishes it from PPO clipping, and gives coefficient examples without deriving KL divergence.
Examples
Uses a medical-support chatbot and contrasts training with and without an anchor using reported KL values.
What you can do after
Apply it by penalizing divergence from a reference policy and adjusting the coefficient to balance improvement and stability.
How it connects
Connects KL regularization to PPO, GRPO, RLHF, reward shaping, and trust-region constraints.

Also explained in

Compare how 5 books explain KL regularization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.