Reverse KL divergence
Reverse Kullback–Leibler divergence is the divergence with its distribution arguments reversed relative to the forward divergence, so the expectation is taken under the first distribution.
Reinforcement Learning from Human Feedback
Reverse KL divergence is the Kullback-Leibler divergence from a target distribution to a reference distribution, penalizing probability mass assigned by the target where the reference assigns little probability.
- How it is used
- Regularizes RL policies against a reference and trains students on their own generated states in on-policy distillation.
- How deep it goes
- Defines KL direction, interprets its sampling and penalties, and derives the relationship between the RL objective and a reward-optimal policy.
- Examples
- Illustrates forgetting with old and new distribution modes and combines math and code specialists in multi-teacher distillation.
- What you can do after
- Evaluate how KL direction changes sampling, penalties, and policy behavior in post-training objectives.
Read how Reinforcement Learning from Human Feedback explains reverse KL divergence →
LLM Customization and Fine-Tuning
The Kullback–Leibler divergence from a student distribution to a teacher distribution.
- How it is used
- Shapes student behavior in teacher–student distillation when focused, precise outputs matter more than broad coverage.
- How deep it goes
- Defines KL divergence and contrasts training behavior, while qualifying textbook mode-seeking intuition with LLM distillation findings.
- Examples
- Contrasts JSON and Markdown generation to illustrate focused output versus broader format coverage.
- What you can do after
- Evaluate whether focused outputs justify reverse KL’s narrower coverage, teacher-access requirements, and higher compute cost.
Read how LLM Customization and Fine-Tuning explains reverse KL divergence →
Also covered in
What to know first
- distillationDistillation is a training technique in which a student model learns to approximate the outputs or behavior of a teacher model.
- probability distributionA probability distribution is a mathematical specification assigning probabilities or probability densities to the possible values or outcomes of a random variable.
- teacher modelA teacher model is a model whose outputs, behavior, or learned representations provide training signals for another model.
Related concepts
- Kullback–Leibler divergencecontrasts with · Kullback–Leibler divergence is a nonnegative, generally asymmetric measure of the difference between probability distributions, defined as the expected log probability ratio under one distribution.
- reference modeluses · A reference model is a fixed policy model whose output distribution provides a baseline for comparing or constraining an optimized policy.
- reinforcement learningequivalent to · Reinforcement learning is a machine-learning paradigm in which an agent learns behavior or a decision policy through interaction with an environment and evaluative feedback, such as rewards.
- mode seekingproduces · Mode seeking is a tendency for a distribution approximation to concentrate probability around one or a few high-probability modes rather than represent the target distribution’s full spread.
- student modelproduces · A student model is a model trained with guidance from a teacher model to approximate its outputs, behavior, or learned capabilities.
Questions about reverse KL divergence
What is reverse KL divergence?
Reverse Kullback–Leibler divergence is the divergence with its distribution arguments reversed relative to the forward divergence, so the expectation is taken under the first distribution.
Which Manning books explain reverse KL divergence?
Reinforcement Learning from Human Feedback and LLM Customization and Fine-Tuning.
What should I know before learning reverse KL divergence?
Manning books that teach reverse KL divergence usually introduce distillation, probability distribution and teacher model first.
What is the difference between reverse KL divergence and Kullback–Leibler divergence?
Reverse Kullback–Leibler divergence is the divergence with its distribution arguments reversed relative to the forward divergence, so the expectation is taken under the first distribution. Kullback–Leibler divergence is a nonnegative, generally asymmetric measure of the difference between probability distributions, defined as the expected log probability ratio under one distribution.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

