Distillation loss trains a student by combining correctness from ground-truth labels with richer class relationships encoded in a teacher’s softened predictions. The hard-label term uses cross-entropy at T=1; the soft-target term uses KL divergence between teacher and student distributions at temperature T. section 8.4.4
Assemble the objective from two complementary signals:
- Compute hard-label loss with standard cross-entropy on the student’s unscaled logits and ground-truth labels. section 8.4.4
- Compute soft-target loss with KL divergence after dividing teacher and student logits by the same temperature. section 8.4.4
- Weight the terms as L = αL_hard + βT²KL(teacher_soft || student_soft), commonly using α=0.3 and β=0.7. section 8.4.4
- Use T² to compensate for gradients reduced by temperature scaling, keeping soft-target training dynamics comparable across temperature values. section 8.4.4
PyTorch implementation
A compact implementation applies softmax to teacher logits, log-softmax to student logits, KL divergence with batchmean reduction, and cross-entropy to the labels. It returns alpha times the hard loss plus (1-alpha) times temperature squared times the KL loss. section 8.6
