Soft-target loss measures the KL divergence between teacher and student probability distributions after dividing their logits by temperature T. It transfers the teacher’s dark knowledge: relative probabilities that encode relationships among classes. section 8.4.4
Compute teacher probabilities with softmax(teacher_logits / T), student log-probabilities with log_softmax(student_logits / T), and compare them with KL divergence. Multiplying the result by T² offsets the gradient reduction caused by temperature scaling. section 8.4.4 section 8.6.2
Use soft-target loss as part of a grounded distillation objective:
- Combine it with hard-label cross-entropy: L = αL_hard + (1 − α)T²L_soft. section 8.4.4
- Keep the teacher frozen and run its forward pass under torch.no_grad(); backpropagate the combined loss through the student only. section 8.6.2
- A common weighting uses α = 0.3 for hard labels and 0.7 for soft targets, preserving ground-truth grounding while emphasizing teacher guidance. section 8.4.4
Numerical example
With teacher logits [5.0, 3.0, 1.0, −1.0], student logits [4.0, 2.5, 1.5, −0.5], and T = 3, the soft-target distributions are [0.523, 0.268, 0.138, 0.071] and [0.442, 0.268, 0.192, 0.099]. Their KL divergence is approximately 0.0199, giving L_soft ≈ 0.179 after T² scaling. With L_hard ≈ 0.275 and α = 0.3, the combined loss is approximately 0.208. section 8.6.2
Shows the compact equation linking hard-label grounding, temperature-scaled soft-target KL divergence, and the combined loss. section 8.4.4
Listing 8.2 The knowledge distillation loss function.
import torch.nn.functional as F
def distillation_loss(student_logits,
[CA] teacher_logits, labels,
[CA] temperature=3.0, alpha=0.3): #A
soft_teacher = F.softmax(
[CA] teacher_logits / temperature,
[CA] dim=-1) #B
soft_student = F.log_softmax(
[CA] student_logits / temperature,
[CA] dim=-1) #C
kl_loss = F.kl_div(
[CA] soft_student, soft_teacher,
[CA] reduction='batchmean') #D
hard_loss = F.cross_entropy(
[CA] student_logits, labels) #E
return (alpha * hard_loss
[CA] + (1 - alpha)
[CA] * temperature**2
[CA] * kl_loss) #FProvides a PyTorch implementation using softmax, log_softmax, KL divergence, cross-entropy, and temperature scaling. section 8.4.4
