liveBook · Manning

KL regularization

As explained in LLM Evaluation and Alignment, The Foundational Ideas

KL control is the use of a Kullback–Leibler divergence penalty or constraint to limit a policy's drift from a reference policy during optimization.

A policy optimized against a reward model can drift toward responses that score highly but do not reflect the intended behavior. KL regularization limits that drift by penalizing differences between the current policy and a reference policy. Kullback–Leibler (KL) divergence measures the difference between their response distributions. section 7.5.4 section 7.5.1

In reinforcement learning from human feedback (RLHF), the reference is often the supervised fine-tuned (SFT) model that also initializes the policy. The optimization trades reward-model gains against the cost of moving away from that reference: adjusted reward = reward-model score − β × KL divergence. The positive coefficient β controls the penalty; one cited implementation uses about 0.02. section 7.5.1 section 7.5.4

Figure 7.8 The InstructGPT three-stage pipeline. Stage 1 fine-tunes the pretrained model on labeler-written demonstrations. Stage 2 trains a reward model on labeler-ranked comparisons of model outputs. Stage 3 optimizes the SFT model with reinforcement learning using PPO, with the reward model's score (penalized by KL divergence from the SFT model) as the reward signal. The SFT model from stage 1 plays two roles: it initializes the RL policy and is the reference distribution in the KL penalty. Source: Figure 2 of Ouyang et al. 2022.

The pipeline shows how the SFT model supplies both the initial policy and the reference for the KL penalty. section 7.5.1

KL regularization and PPO clipping serve different roles. PPO clips individual policy updates to keep the updated policy close to the previous one; the KL penalty discourages drift from the SFT reference. Both help limit unstable optimization, but neither makes the reward model a perfect measure of quality. section 7.5.4 section 7.8

A guardrail with a tradeoff

A reward model may overvalue numbered lists or hedging, leading the policy to use them even when they do not improve an answer. A KL penalty can restrain this overoptimization, but it also limits useful behavioral changes. Track reward-model scores against held-out human preferences: rising scores alongside flat or falling human ratings signal possible reward hacking. section 7.6.1

KL regularization also appears with group relative policy optimization (GRPO) and reference-based statistical rewards. In that setting, constrained updates are part of a broader setup that includes a strong base model, curated prompts, and high-quality reference answers; the penalty alone does not make a fixed metric reliable. section 7.7.4

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains the reward tradeoff and distinguishes KL penalty from PPO clipping, while deferring training and optimizer details.
Examples
Uses an SFT reference in InstructGPT and describes GRPO with reference-based statistical rewards.
What you can do after
Diagnose potential reward hacking by comparing reward-model scores with held-out human preferences during training.
How it connects
Connects KL divergence with supervised fine-tuning, reward modeling, PPO, and reward hacking.

Also explained in

Compare how 5 books explain KL regularization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.