Kullback–Leibler (KL) divergence measures the discrepancy between two probability distributions. In model training, it can pull a student toward a teacher’s predictions or discourage a policy from drifting too far from a reference policy. section 8.4.4 section 7.4.2
In classical knowledge distillation, the teacher and student process the same input. Their logits are divided by temperature T and converted to soft probability distributions. The soft loss uses KL(teacher || student), transferring information about the teacher’s relative probabilities for different classes. A separate cross-entropy loss keeps the student grounded in the correct labels. section 8.5.3 section 8.4.4
Higher temperatures flatten both distributions and reduce the soft-loss gradient. Multiplying KL by T² compensates for the approximate 1/T² gradient scaling. The combined loss weights this scaled term against hard-label cross-entropy; a common choice assigns 0.7 to the soft term and 0.3 to the hard term. section 8.3.4 section 8.4.4
Listing 8.2 The knowledge distillation loss function.
import torch.nn.functional as F
def distillation_loss(student_logits,
[CA] teacher_logits, labels,
[CA] temperature=3.0, alpha=0.3): #A
soft_teacher = F.softmax(
[CA] teacher_logits / temperature,
[CA] dim=-1) #B
soft_student = F.log_softmax(
[CA] student_logits / temperature,
[CA] dim=-1) #C
kl_loss = F.kl_div(
[CA] soft_student, soft_teacher,
[CA] reduction='batchmean') #D
hard_loss = F.cross_entropy(
[CA] student_logits, labels) #E
return (alpha * hard_loss
[CA] + (1 - alpha)
[CA] * temperature**2
[CA] * kl_loss) #FThe PyTorch loss applies softmax to teacher logits, log-softmax to student logits, and F.kl_div with batch-mean reduction before adding the weighted hard-label loss. section 8.4.4
A temperature-scaled loss
At T=3, teacher probabilities [0.523, 0.268, 0.138, 0.071] and student probabilities [0.442, 0.268, 0.192, 0.099] give KL approximately 0.0199. Multiplying by T² gives a soft loss of about 0.179. Combined with hard-label loss 0.275 at weights 0.7 and 0.3, the total is approximately 0.208. section 8.6.2
Not all distillation uses KL. An alternative trains the student on curated teacher-generated responses with next-token prediction. It avoids simultaneous teacher and student inference during student training, but transfers generated text rather than the teacher’s probability distribution. section 8.5.3
In GRPO, KL serves a different purpose: it penalizes movement of the current policy away from a frozen reference policy. The clipped ratio instead compares the current policy with the old policy that generated the completions. Together with group-relative advantages, these pressures shape the update; even a positively rewarded token can be constrained by reference-policy KL. section 7.4.2 section 7.9.1
The full GRPO objective subtracts reference-policy KL from the clipped, advantage-weighted term. section 7.4.2
Monitor KL alongside reward, pass rate, response length, and other task-specific metrics. Rising reward alone can hide excessive policy drift or degraded behavior. section 7.10.2


















