Kullback–Leibler (KL) divergence measures how one probability distribution differs from a reference distribution. It is nonnegative and asymmetric: DKL(P || Q) generally differs from DKL(Q || P), and equals zero only when the distributions match. section 7.5
In knowledge distillation, KL divergence describes how a student model matches a teacher’s output distribution. Standard supervised fine-tuning on sampled teacher answers is the practical forward-KL default. Reverse KL needs the teacher’s per-token probabilities, so it requires white-box access to weights and logits. section 7.5
Choose the divergence to match the deployment tradeoff:
- Forward KL emphasizes coverage: it penalizes the student for giving too little probability to tokens the teacher considers likely. The student may cover more patterns but spread capacity thin. section 7.5.1
- Reverse KL emphasizes precision: it penalizes probability on tokens the teacher considers unlikely. The student can become more focused, while dropping some of the teacher’s capabilities. section 7.5.1
- For most enterprise distillation, start with forward KL through standard SFT. Consider reverse KL for tasks where narrow, correct outputs matter more than broad coverage, and only when teacher-scoring infrastructure is available. section 7.5.3
Structured output
Suppose a teacher generates both JSON and Markdown for data extraction. Forward KL may teach the student both formats, sometimes producing an invalid hybrid. Reverse KL may focus it on the teacher’s most confident format—JSON—yielding narrower but more consistent output. section 7.5.1
Table 7.5 Divergence measures for knowledge distillation
|
Approach
|
Training signal
|
Student behavior
|
Compute cost
|
Tooling
|
|
Forward KL (standard SFT)
|
Teacher text outputs
|
Broad coverage, may spread
thin
|
1x (baseline)
|
SFTTrainer,
native
|
|
Reverse KL (on-policy)
|
Student generates, teacher
scores
|
Precise, focused on the
teacher’s strengths
|
2 to 3x
|
Custom training loop
|
|
Adaptive KL
|
Dynamic blend
|
Balanced coverage and precision
|
1.5 to 2x
|
Research code only
|
|
Selective token weighting
|
Weighted by token informativeness
|
Strong on distinctive patterns
|
1.2 to 1.5x
|
Custom loss function
|
Compare forward, reverse, adaptive, and selectively weighted approaches by training signal, student behavior, compute cost, and tooling. section 7.5.3


















