Kullback–Leibler (KL) divergence measures how a probability distribution differs from a reference distribution. In knowledge distillation, it provides a loss for aligning a student model’s output probabilities with a teacher’s: lower divergence indicates closer agreement. section 2.5
A logits-only distillation cycle uses KL divergence to correct the student:
- Take a batch of tokenized text and obtain teacher logits without tracking gradients. section 2.5
- Run the student on the same inputs. Convert its logits with log-softmax and the teacher’s with softmax, then compute KL loss. section 2.5
- Backpropagate the loss, update the student’s weights, and clear gradients before the next batch. section 2.5
Listing 2.5 Step 4: Correcting the attempt
loss = F.kl_div( F.log_softmax(student_logits, dim=-1), F.softmax(teacher_logits, dim=-1), reduction='batchmean' ) loss.backward() #A optimizer.step() #B optimizer.zero_grad() #C
The correction step computes KL loss with PyTorch and applies an optimizer update. section 2.5
For a compound loss, KL divergence can supply the logit-matching signal alongside next-token task loss and hidden-state alignment. A baseline weights these components 0.4, 0.4, and 0.2 respectively; the weights control their relative influence on training. section 6.3.1
student_soft = F.log_softmax(
student_logits / temperature,
dim=-1) #A
teacher_soft = F.softmax(
teacher_logits / temperature,
dim=-1) #B
student_soft = student_soft[..., :-1, :].contiguous()
teacher_soft = teacher_soft[..., :-1, :].contiguous()
loss_logits = F.kl_div(
student_soft.view(-1, student_soft.size(-1)),
teacher_soft.view(-1, teacher_soft.size(-1)),
reduction='batchmean' #C
) * (temperature ** 2) #D
The logit-loss implementation divides both models’ logits by temperature, removes the last sequence position, flattens the remaining positions, and scales KL loss by temperature squared. section 6.3.1
Temperature softens the teacher’s probability distribution, making alternatives more visible to the student. The compound-loss implementation uses a default temperature of 2.0. section 6.3.1


















