liveBook · Manning

Reverse KL divergence

As explained in LLM Customization and Fine-Tuning

The Kullback–Leibler divergence from a student distribution to a teacher distribution.

Reverse KL divergence measures the difference between a student’s probability distribution and a teacher’s, with the student distribution as the reference. In distillation, it penalizes the student for assigning high probability to tokens the teacher considers unlikely. section 7.5 section 7.5.1

Using reverse KL requires the teacher’s per-token probabilities across the vocabulary, so it needs white-box access to the teacher’s weights and logits. In on-policy training, the student generates tokens for the teacher to score; this requires a custom training loop and roughly 2–3 times the compute of standard SFT. section 7.5 section 7.5.3

Use the precision-versus-coverage tradeoff to guide the choice:

  • Reverse KL favors focused outputs and can ignore some teacher capabilities; forward KL favors broader coverage. section 7.5.1
  • Consider reverse KL when retention is below 80%, more data has not helped, or precise outputs matter more than broad capability. section 7.5.3
  • Treat mode-seeking as an intuition, not a guaranteed outcome: research on LLM distillation finds the distinction less clean, with practical differences tied to head-versus-tail focus early in training. section 7.5.1

Structured output

If a teacher produces both JSON and Markdown, reverse KL can lead the student to favor the format the teacher uses most confidently. A student that consistently produces valid JSON may be preferable when deployment requires that format, even if it has narrower capabilities. section 7.5.1

Table 7.5 Divergence measures for knowledge distillation
Approach
Training signal
Student behavior
Compute cost
Tooling
Forward KL (standard SFT)
Teacher text outputs
Broad coverage, may spread thin
1x (baseline)
SFTTrainer, native
Reverse KL (on-policy)
Student generates, teacher scores
Precise, focused on the teacher’s strengths
2 to 3x
Custom training loop
Adaptive KL
Dynamic blend
Balanced coverage and precision
1.5 to 2x
Research code only
Selective token weighting
Weighted by token informativeness
Strong on distinctive patterns
1.2 to 1.5x
Custom loss function

Compare divergence choices by training signal, student behavior, compute cost, and tooling. section 7.5.3

How LLM Customization and Fine-Tuning covers it

How deep it goes
Defines KL divergence and contrasts training behavior, while qualifying textbook mode-seeking intuition with LLM distillation findings.
Examples
Contrasts JSON and Markdown generation to illustrate focused output versus broader format coverage.
What it assumes
Defines KL divergence from its distribution-matching role but requires white-box teacher access and per-token probabilities for optimization.
What you can do after
Evaluate whether focused outputs justify reverse KL’s narrower coverage, teacher-access requirements, and higher compute cost.
How it connects
Connects knowledge distillation with forward KL, supervised fine-tuning, and student–teacher probability distributions.

Also explained in

Compare how 2 books explain reverse KL divergence →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.