KL regularization
KL regularization is a penalty or constraint based on Kullback–Leibler divergence that limits how far a learned distribution can deviate from a reference distribution.
Applied Reinforcement Learning
A regularization term based on Kullback–Leibler divergence that discourages a policy from moving too far from a reference policy.
- How it is used
- Constrains policy drift from a reference while optimizing reward in PPO and GRPO training.
- How deep it goes
- Explains the reward-penalty role, distinguishes it from PPO clipping, and gives coefficient examples without deriving KL divergence.
- Examples
- Uses a medical-support chatbot and contrasts training with and without an anchor using reported KL values.
- What you can do after
- Apply it by penalizing divergence from a reference policy and adjusting the coefficient to balance improvement and stability.
Read how Applied Reinforcement Learning explains KL regularization →
Reinforcement Learning from Human Feedback
A KL penalty is a regularization term proportional to the Kullback–Leibler divergence between an optimized distribution or policy and a reference distribution or policy.
- How it is used
- Constrains policy drift while optimizing reward during RLHF fine-tuning.
- How deep it goes
- Derives the regularized objective and discusses penalty placement, KL estimation, controllers, and interaction with PPO clipping.
- Examples
- Uses GRPO with multiple completions per prompt to illustrate tokenwise KL and loss placement.
- What you can do after
- Operate/configure a KL penalty by choosing its strength and placement, and monitor divergence during training.
Read how Reinforcement Learning from Human Feedback explains KL regularization →
LLM Evaluation and Alignment, The Foundational Ideas
KL control is the use of a Kullback–Leibler divergence penalty or constraint to limit a policy's drift from a reference policy during optimization.
- How it is used
- Constrains policy drift from a reference while optimizing reward in RLHF.
- How deep it goes
- Explains the reward tradeoff and distinguishes KL penalty from PPO clipping, while deferring training and optimizer details.
- Examples
- Uses an SFT reference in InstructGPT and describes GRPO with reference-based statistical rewards.
- What you can do after
- Diagnose potential reward hacking by comparing reward-model scores with held-out human preferences during training.
Read how LLM Evaluation and Alignment, The Foundational Ideas explains KL regularization →
Build a DeepSeek Model (From Scratch)
KL control is the regulation of policy updates by penalizing divergence between the learned policy and a reference policy.
- How it is used
- Stabilizes PPO- and GRPO-style language-model reinforcement learning by constraining updates against a frozen reference policy.
- How deep it goes
- Explains the frozen anchor, token-level log-probability comparison, masking, monitoring, and pressure trade-offs, but not a full optimizer implementation.
- Examples
- Uses frozen pre-training copies, generated response tokens, attention masks, log probabilities, and reward-KL diagnostic metrics.
- What you can do after
- Operate/configure KL control by freezing a reference policy, masking tokens, computing divergence, tuning pressure, and monitoring stability.
Read how Build a DeepSeek Model (From Scratch) explains KL regularization →
LLM Customization and Fine-Tuning
A constraint based on Kullback–Leibler divergence that limits how far a policy distribution can move from a reference distribution.
- How it is used
- Constrains DPO preference updates to limit drift from the pre-DPO model.
- How deep it goes
- Explains the reference-model role and beta effects, without deriving the KL divergence mathematically.
- Examples
- Uses a TRL beta baseline of 0.1 and describes risks below 0.05 and above 0.3.
- What you can do after
- Configure DPO’s KL strength through beta in TRL’s DPOTrainer.
Read how LLM Customization and Fine-Tuning explains KL regularization →
What to know first
- reward modelA reward model is a learned model that assigns scores to outputs, actions, or outcomes to represent their desirability according to preferences or evaluation criteria.
- Kullback–Leibler divergenceKullback–Leibler divergence is a nonnegative, generally asymmetric measure of the difference between probability distributions, defined as the expected log probability ratio under one distribution.
- policyA policy is a rule, principle, or decision function that governs behavior or determines actions or permissions under specified conditions.
- Proximal Policy OptimizationProximal Policy Optimization is a policy-gradient reinforcement-learning algorithm that limits policy updates to keep successive policies close and promote stable learning.
Related concepts
- Group Relative Policy Optimizationused by · Group Relative Policy Optimization is a reinforcement-learning algorithm that updates a policy using relative rewards or advantages estimated from groups of sampled actions or outputs.
- Proximal Policy Optimizationused by · Proximal Policy Optimization is a policy-gradient reinforcement-learning algorithm that limits policy updates to keep successive policies close and promote stable learning.
- reference modeluses · A reference model is a fixed policy model whose output distribution provides a baseline for comparing or constraining an optimized policy.
- Kullback–Leibler divergenceuses · Kullback–Leibler divergence is a nonnegative, generally asymmetric measure of the difference between probability distributions, defined as the expected log probability ratio under one distribution.
- DPO losspart of · The DPO loss is a preference-training objective that optimizes a policy to assign higher relative likelihood to preferred responses than to rejected responses compared with a reference policy.
- RLHFused by · Reinforcement learning from human feedback is a model-training approach that uses human evaluations or preferences as feedback to optimize model behavior.
Questions about KL regularization
What is KL regularization?
KL regularization is a penalty or constraint based on Kullback–Leibler divergence that limits how far a learned distribution can deviate from a reference distribution.
Which Manning books explain KL regularization?
Applied Reinforcement Learning, Reinforcement Learning from Human Feedback, LLM Evaluation and Alignment, The Foundational Ideas, Build a DeepSeek Model (From Scratch) and LLM Customization and Fine-Tuning.
What should I know before learning KL regularization?
Manning books that teach KL regularization usually introduce reward model, Kullback–Leibler divergence, policy and Proximal Policy Optimization first.
What is KL regularization part of?
KL regularization is part of DPO loss. The DPO loss is a preference-training objective that optimizes a policy to assign higher relative likelihood to preferred responses than to rejected responses compared with a reference policy.
What is KL regularization also called?
KL regularization is also called KL penalty or KL constraint.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.




