liveBook · Manning

Kullback–Leibler divergence

As explained in LLM Customization and Fine-Tuning

A measure of how one probability distribution differs from a reference probability distribution.

Kullback–Leibler (KL) divergence measures how one probability distribution differs from a reference distribution. It is nonnegative and asymmetric: DKL(P || Q) generally differs from DKL(Q || P), and equals zero only when the distributions match. section 7.5

In knowledge distillation, KL divergence describes how a student model matches a teacher’s output distribution. Standard supervised fine-tuning on sampled teacher answers is the practical forward-KL default. Reverse KL needs the teacher’s per-token probabilities, so it requires white-box access to weights and logits. section 7.5

Choose the divergence to match the deployment tradeoff:

  • Forward KL emphasizes coverage: it penalizes the student for giving too little probability to tokens the teacher considers likely. The student may cover more patterns but spread capacity thin. section 7.5.1
  • Reverse KL emphasizes precision: it penalizes probability on tokens the teacher considers unlikely. The student can become more focused, while dropping some of the teacher’s capabilities. section 7.5.1
  • For most enterprise distillation, start with forward KL through standard SFT. Consider reverse KL for tasks where narrow, correct outputs matter more than broad coverage, and only when teacher-scoring infrastructure is available. section 7.5.3

Structured output

Suppose a teacher generates both JSON and Markdown for data extraction. Forward KL may teach the student both formats, sometimes producing an invalid hybrid. Reverse KL may focus it on the teacher’s most confident format—JSON—yielding narrower but more consistent output. section 7.5.1

Table 7.5 Divergence measures for knowledge distillation
Approach
Training signal
Student behavior
Compute cost
Tooling
Forward KL (standard SFT)
Teacher text outputs
Broad coverage, may spread thin
1x (baseline)
SFTTrainer, native
Reverse KL (on-policy)
Student generates, teacher scores
Precise, focused on the teacher’s strengths
2 to 3x
Custom training loop
Adaptive KL
Dynamic blend
Balanced coverage and precision
1.5 to 2x
Research code only
Selective token weighting
Weighted by token informativeness
Strong on distinctive patterns
1.2 to 1.5x
Custom loss function

Compare forward, reverse, adaptive, and selectively weighted approaches by training signal, student behavior, compute cost, and tooling. section 7.5.3

How LLM Customization and Fine-Tuning covers it

How deep it goes
Defines nonnegative, asymmetric divergence and explains its training effects without deriving the measure.
Examples
Contrasts student behavior on a teacher that produces both JSON and Markdown outputs.
What you can do after
Evaluate forward or reverse KL by balancing output coverage, precision, task requirements, and teacher access.
How it connects
Connects knowledge distillation with supervised fine-tuning, cross-entropy loss, and teacher–student models.

Also explained in

Compare how 19 books explain Kullback–Leibler divergence →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.