liveBook · Manning

Kullback–Leibler divergence

As explained in Rearchitecting LLMs

Kullback–Leibler divergence measures how one probability distribution differs from a reference distribution.

Kullback–Leibler (KL) divergence measures how a probability distribution differs from a reference distribution. In knowledge distillation, it provides a loss for aligning a student model’s output probabilities with a teacher’s: lower divergence indicates closer agreement. section 2.5

A logits-only distillation cycle uses KL divergence to correct the student:

  • Take a batch of tokenized text and obtain teacher logits without tracking gradients. section 2.5
  • Run the student on the same inputs. Convert its logits with log-softmax and the teacher’s with softmax, then compute KL loss. section 2.5
  • Backpropagate the loss, update the student’s weights, and clear gradients before the next batch. section 2.5
Listing 2.5 Step 4: Correcting the attempt
loss = F.kl_div(
   F.log_softmax(student_logits, dim=-1),
   F.softmax(teacher_logits, dim=-1),
   reduction='batchmean'
)
loss.backward()      #A
optimizer.step()     #B
optimizer.zero_grad()  #C
 

The correction step computes KL loss with PyTorch and applies an optimizer update. section 2.5

For a compound loss, KL divergence can supply the logit-matching signal alongside next-token task loss and hidden-state alignment. A baseline weights these components 0.4, 0.4, and 0.2 respectively; the weights control their relative influence on training. section 6.3.1

student_soft = F.log_softmax(
    student_logits / temperature, 
    dim=-1) #A
teacher_soft = F.softmax(
    teacher_logits / temperature, 
    dim=-1) #B
 
student_soft = student_soft[..., :-1, :].contiguous()
teacher_soft = teacher_soft[..., :-1, :].contiguous()
 
loss_logits = F.kl_div(
    student_soft.view(-1, student_soft.size(-1)),
    teacher_soft.view(-1, teacher_soft.size(-1)),
    reduction='batchmean' #C
) * (temperature ** 2) #D

The logit-loss implementation divides both models’ logits by temperature, removes the last sequence position, flattens the remaining positions, and scales KL loss by temperature squared. section 6.3.1

Temperature softens the teacher’s probability distribution, making alternatives more visible to the student. The compound-loss implementation uses a default temperature of 2.0. section 6.3.1

How Rearchitecting LLMs covers it

How deep it goes
Defines divergence through distribution agreement and implements both direct and temperature-scaled KL loss, without deriving the formula.
Examples
Uses a pruned student matching a teacher’s logits and a compound loss combining task, logit, and hidden-state signals.
What you can do after
Apply it via PyTorch to train a student against teacher probabilities, either directly or as part of a compound loss.
How it connects
Connects KL divergence to knowledge distillation, probability distributions, softmax, temperature scaling, and compound loss.

Also explained in

Compare how 19 books explain Kullback–Leibler divergence →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.