liveBook · Manning

Distillation loss

As explained in Build a DeepSeek Model (From Scratch)

A distillation loss is a training objective that measures or combines discrepancies between a student's predictions and information supplied by a teacher model.

Distillation loss trains a student by combining correctness from ground-truth labels with richer class relationships encoded in a teacher’s softened predictions. The hard-label term uses cross-entropy at T=1; the soft-target term uses KL divergence between teacher and student distributions at temperature T. section 8.4.4

Assemble the objective from two complementary signals:

  • Compute hard-label loss with standard cross-entropy on the student’s unscaled logits and ground-truth labels. section 8.4.4
  • Compute soft-target loss with KL divergence after dividing teacher and student logits by the same temperature. section 8.4.4
  • Weight the terms as L = αL_hard + βT²KL(teacher_soft || student_soft), commonly using α=0.3 and β=0.7. section 8.4.4
  • Use T² to compensate for gradients reduced by temperature scaling, keeping soft-target training dynamics comparable across temperature values. section 8.4.4

PyTorch implementation

A compact implementation applies softmax to teacher logits, log-softmax to student logits, KL divergence with batchmean reduction, and cross-entropy to the labels. It returns alpha times the hard loss plus (1-alpha) times temperature squared times the KL loss. section 8.6

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains the two loss terms, temperature-gradient compensation, weighting, and PyTorch operations, then applies them in a training pipeline.
Examples
Uses a weighted equation, a PyTorch loss function, toy teacher-student models, and comparisons against hard-label training and teacher performance.
What it assumes
Assumes familiarity with logits, softmax, cross-entropy, KL divergence, and PyTorch while grounding the teacher-student workflow.
What you can do after
Build it from scratch by implementing the loss and integrating it into a runnable PyTorch distillation pipeline.
How it connects
Connects knowledge distillation with soft targets, hard labels, cross-entropy, KL divergence, temperature scaling, and student-teacher training.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.