liveBook · Manning

Kullback–Leibler divergence

As explained in Build a DeepSeek Model (From Scratch)

KL divergence is a measure of the discrepancy between two probability distributions that quantifies information lost when one distribution approximates another.

Kullback–Leibler (KL) divergence measures the discrepancy between two probability distributions. In model training, it can pull a student toward a teacher’s predictions or discourage a policy from drifting too far from a reference policy. section 8.4.4 section 7.4.2

In classical knowledge distillation, the teacher and student process the same input. Their logits are divided by temperature T and converted to soft probability distributions. The soft loss uses KL(teacher || student), transferring information about the teacher’s relative probabilities for different classes. A separate cross-entropy loss keeps the student grounded in the correct labels. section 8.5.3 section 8.4.4

Higher temperatures flatten both distributions and reduce the soft-loss gradient. Multiplying KL by T² compensates for the approximate 1/T² gradient scaling. The combined loss weights this scaled term against hard-label cross-entropy; a common choice assigns 0.7 to the soft term and 0.3 to the hard term. section 8.3.4 section 8.4.4

Listing 8.2 The knowledge distillation loss function.
import torch.nn.functional as F
def distillation_loss(student_logits,
     [CA] teacher_logits, labels,
     [CA] temperature=3.0, alpha=0.3):  #A
    soft_teacher = F.softmax(
     [CA] teacher_logits / temperature,
     [CA] dim=-1)  #B
    soft_student = F.log_softmax(
     [CA] student_logits / temperature,
     [CA] dim=-1)  #C
    kl_loss = F.kl_div(
     [CA] soft_student, soft_teacher,
     [CA] reduction='batchmean')  #D
 
    hard_loss = F.cross_entropy(
     [CA] student_logits, labels)  #E
    return (alpha * hard_loss
     [CA] + (1 - alpha)
     [CA] * temperature**2
     [CA] * kl_loss)  #F

The PyTorch loss applies softmax to teacher logits, log-softmax to student logits, and F.kl_div with batch-mean reduction before adding the weighted hard-label loss. section 8.4.4

A temperature-scaled loss

At T=3, teacher probabilities [0.523, 0.268, 0.138, 0.071] and student probabilities [0.442, 0.268, 0.192, 0.099] give KL approximately 0.0199. Multiplying by T² gives a soft loss of about 0.179. Combined with hard-label loss 0.275 at weights 0.7 and 0.3, the total is approximately 0.208. section 8.6.2

Not all distillation uses KL. An alternative trains the student on curated teacher-generated responses with next-token prediction. It avoids simultaneous teacher and student inference during student training, but transfers generated text rather than the teacher’s probability distribution. section 8.5.3

In GRPO, KL serves a different purpose: it penalizes movement of the current policy away from a frozen reference policy. The clipped ratio instead compares the current policy with the old policy that generated the completions. Together with group-relative advantages, these pressures shape the update; even a positively rewarded token can be constrained by reference-policy KL. section 7.4.2 section 7.9.1

LGRPO(θ) = E[min(ratioₖ(θ)Aᵢ, clip(ratioₖ(θ), 1-ε, 1+ε)Aᵢ) - β KL(πθ || πref)]

The full GRPO objective subtracts reference-policy KL from the clipped, advantage-weighted term. section 7.4.2

Monitor KL alongside reward, pass rate, response length, and other task-specific metrics. Rising reward alone can hide excessive policy drift or degraded behavior. section 7.10.2

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains temperature-corrected gradients and a PyTorch loss, while leaving production RL details such as KL scheduling to further work.
Examples
Works through a four-class distillation loss at T=3 and a GRPO objective with a reference-policy penalty.
What you can do after
Apply it via PyTorch to combine teacher-student KL with hard-label cross-entropy.
How it connects
Connects knowledge distillation, softmax temperature, cross-entropy, policy gradients, and clipped objectives.

Also explained in

Compare how 19 books explain Kullback–Leibler divergence →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.