Reverse KL divergence measures the difference between a student’s probability distribution and a teacher’s, with the student distribution as the reference. In distillation, it penalizes the student for assigning high probability to tokens the teacher considers unlikely. section 7.5 section 7.5.1
Using reverse KL requires the teacher’s per-token probabilities across the vocabulary, so it needs white-box access to the teacher’s weights and logits. In on-policy training, the student generates tokens for the teacher to score; this requires a custom training loop and roughly 2–3 times the compute of standard SFT. section 7.5 section 7.5.3
Use the precision-versus-coverage tradeoff to guide the choice:
- Reverse KL favors focused outputs and can ignore some teacher capabilities; forward KL favors broader coverage. section 7.5.1
- Consider reverse KL when retention is below 80%, more data has not helped, or precise outputs matter more than broad capability. section 7.5.3
- Treat mode-seeking as an intuition, not a guaranteed outcome: research on LLM distillation finds the distinction less clean, with practical differences tied to head-versus-tail focus early in training. section 7.5.1
Structured output
If a teacher produces both JSON and Markdown, reverse KL can lead the student to favor the format the teacher uses most confidently. A student that consistently produces valid JSON may be preferable when deployment requires that format, even if it has narrower capabilities. section 7.5.1
Table 7.5 Divergence measures for knowledge distillation
|
Approach
|
Training signal
|
Student behavior
|
Compute cost
|
Tooling
|
|
Forward KL (standard SFT)
|
Teacher text outputs
|
Broad coverage, may spread
thin
|
1x (baseline)
|
SFTTrainer,
native
|
|
Reverse KL (on-policy)
|
Student generates, teacher
scores
|
Precise, focused on the
teacher’s strengths
|
2 to 3x
|
Custom training loop
|
|
Adaptive KL
|
Dynamic blend
|
Balanced coverage and precision
|
1.5 to 2x
|
Research code only
|
|
Selective token weighting
|
Weighted by token informativeness
|
Strong on distinctive patterns
|
1.2 to 1.5x
|
Custom loss function
|
Compare divergence choices by training signal, student behavior, compute cost, and tooling. section 7.5.3

