liveBook · Manning

KL regularization

Also called KL penalty, KL constraint

KL regularization is a penalty or constraint based on Kullback–Leibler divergence that limits how far a learned distribution can deviate from a reference distribution.

How 5 Manning books explain it

Applied Reinforcement Learning

A regularization term based on Kullback–Leibler divergence that discourages a policy from moving too far from a reference policy.

How it is used
Constrains policy drift from a reference while optimizing reward in PPO and GRPO training.
How deep it goes
Explains the reward-penalty role, distinguishes it from PPO clipping, and gives coefficient examples without deriving KL divergence.
Examples
Uses a medical-support chatbot and contrasts training with and without an anchor using reported KL values.
What you can do after
Apply it by penalizing divergence from a reference policy and adjusting the coefficient to balance improvement and stability.

Read how Applied Reinforcement Learning explains KL regularization →

Reinforcement Learning from Human Feedback

A KL penalty is a regularization term proportional to the Kullback–Leibler divergence between an optimized distribution or policy and a reference distribution or policy.

How it is used
Constrains policy drift while optimizing reward during RLHF fine-tuning.
How deep it goes
Derives the regularized objective and discusses penalty placement, KL estimation, controllers, and interaction with PPO clipping.
Examples
Uses GRPO with multiple completions per prompt to illustrate tokenwise KL and loss placement.
What you can do after
Operate/configure a KL penalty by choosing its strength and placement, and monitor divergence during training.

Read how Reinforcement Learning from Human Feedback explains KL regularization →

LLM Evaluation and Alignment, The Foundational Ideas

KL control is the use of a Kullback–Leibler divergence penalty or constraint to limit a policy's drift from a reference policy during optimization.

How it is used
Constrains policy drift from a reference while optimizing reward in RLHF.
How deep it goes
Explains the reward tradeoff and distinguishes KL penalty from PPO clipping, while deferring training and optimizer details.
Examples
Uses an SFT reference in InstructGPT and describes GRPO with reference-based statistical rewards.
What you can do after
Diagnose potential reward hacking by comparing reward-model scores with held-out human preferences during training.

Read how LLM Evaluation and Alignment, The Foundational Ideas explains KL regularization →

Build a DeepSeek Model (From Scratch)

KL control is the regulation of policy updates by penalizing divergence between the learned policy and a reference policy.

How it is used
Stabilizes PPO- and GRPO-style language-model reinforcement learning by constraining updates against a frozen reference policy.
How deep it goes
Explains the frozen anchor, token-level log-probability comparison, masking, monitoring, and pressure trade-offs, but not a full optimizer implementation.
Examples
Uses frozen pre-training copies, generated response tokens, attention masks, log probabilities, and reward-KL diagnostic metrics.
What you can do after
Operate/configure KL control by freezing a reference policy, masking tokens, computing divergence, tuning pressure, and monitoring stability.

Read how Build a DeepSeek Model (From Scratch) explains KL regularization →

LLM Customization and Fine-Tuning

A constraint based on Kullback–Leibler divergence that limits how far a policy distribution can move from a reference distribution.

How it is used
Constrains DPO preference updates to limit drift from the pre-DPO model.
How deep it goes
Explains the reference-model role and beta effects, without deriving the KL divergence mathematically.
Examples
Uses a TRL beta baseline of 0.1 and describes risks below 0.05 and above 0.3.
What you can do after
Configure DPO’s KL strength through beta in TRL’s DPOTrainer.

Read how LLM Customization and Fine-Tuning explains KL regularization →

What to know first

Related concepts

Questions about KL regularization

What is KL regularization?

KL regularization is a penalty or constraint based on Kullback–Leibler divergence that limits how far a learned distribution can deviate from a reference distribution.

Which Manning books explain KL regularization?

Applied Reinforcement Learning, Reinforcement Learning from Human Feedback, LLM Evaluation and Alignment, The Foundational Ideas, Build a DeepSeek Model (From Scratch) and LLM Customization and Fine-Tuning.

What should I know before learning KL regularization?

Manning books that teach KL regularization usually introduce reward model, Kullback–Leibler divergence, policy and Proximal Policy Optimization first.

What is KL regularization part of?

KL regularization is part of DPO loss. The DPO loss is a preference-training objective that optimizes a policy to assign higher relative likelihood to preferred responses than to rejected responses compared with a reference policy.

What is KL regularization also called?

KL regularization is also called KL penalty or KL constraint.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.