liveBook · Manning

Reverse KL divergence

Reverse Kullback–Leibler divergence is the divergence with its distribution arguments reversed relative to the forward divergence, so the expectation is taken under the first distribution.

How 2 Manning books explain it

Reinforcement Learning from Human Feedback

Reverse KL divergence is the Kullback-Leibler divergence from a target distribution to a reference distribution, penalizing probability mass assigned by the target where the reference assigns little probability.

How it is used
Regularizes RL policies against a reference and trains students on their own generated states in on-policy distillation.
How deep it goes
Defines KL direction, interprets its sampling and penalties, and derives the relationship between the RL objective and a reward-optimal policy.
Examples
Illustrates forgetting with old and new distribution modes and combines math and code specialists in multi-teacher distillation.
What you can do after
Evaluate how KL direction changes sampling, penalties, and policy behavior in post-training objectives.

Read how Reinforcement Learning from Human Feedback explains reverse KL divergence →

LLM Customization and Fine-Tuning

The Kullback–Leibler divergence from a student distribution to a teacher distribution.

How it is used
Shapes student behavior in teacher–student distillation when focused, precise outputs matter more than broad coverage.
How deep it goes
Defines KL divergence and contrasts training behavior, while qualifying textbook mode-seeking intuition with LLM distillation findings.
Examples
Contrasts JSON and Markdown generation to illustrate focused output versus broader format coverage.
What you can do after
Evaluate whether focused outputs justify reverse KL’s narrower coverage, teacher-access requirements, and higher compute cost.

Read how LLM Customization and Fine-Tuning explains reverse KL divergence →

Also covered in

What to know first

Related concepts

Questions about reverse KL divergence

What is reverse KL divergence?

Reverse Kullback–Leibler divergence is the divergence with its distribution arguments reversed relative to the forward divergence, so the expectation is taken under the first distribution.

Which Manning books explain reverse KL divergence?

Reinforcement Learning from Human Feedback and LLM Customization and Fine-Tuning.

What should I know before learning reverse KL divergence?

Manning books that teach reverse KL divergence usually introduce distillation, probability distribution and teacher model first.

What is the difference between reverse KL divergence and Kullback–Leibler divergence?

Reverse Kullback–Leibler divergence is the divergence with its distribution arguments reversed relative to the forward divergence, so the expectation is taken under the first distribution. Kullback–Leibler divergence is a nonnegative, generally asymmetric measure of the difference between probability distributions, defined as the expected log probability ratio under one distribution.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.