Hard-label loss is the standard cross-entropy between a student's predictions at temperature T=1 and ground-truth labels. It anchors knowledge distillation in correctness while the soft-target term transfers the teacher's inter-class relationships. section 8.4.4
Use hard-label loss as one branch of the distillation objective:
- Compute cross-entropy directly from the student's raw logits and labels; do not temperature-scale this term. section 8.6.2
- Combine it with the temperature-scaled soft-target loss: L = αL_hard + (1−α)T²L_soft. section 8.4.4 section 8.6.2
- Treat it as a safety net when teacher predictions contain errors or biases; a common weighting uses α=0.3. section 8.4.4
Numerical loss combination
For student logits [4.0, 2.5, 1.5, −0.5] and true class index 0, standard cross-entropy gives L_hard ≈ 0.275. With T=3, the soft loss is approximately 0.179; α=0.3 produces a combined loss of about 0.208. section 8.6.2
Use the complete objective to see how hard-label cross-entropy supplies grounding alongside teacher-guided KL divergence. section 8.4.4
Listing 8.2 The knowledge distillation loss function.
import torch.nn.functional as F
def distillation_loss(student_logits,
[CA] teacher_logits, labels,
[CA] temperature=3.0, alpha=0.3): #A
soft_teacher = F.softmax(
[CA] teacher_logits / temperature,
[CA] dim=-1) #B
soft_student = F.log_softmax(
[CA] student_logits / temperature,
[CA] dim=-1) #C
kl_loss = F.kl_div(
[CA] soft_student, soft_teacher,
[CA] reduction='batchmean') #D
hard_loss = F.cross_entropy(
[CA] student_logits, labels) #E
return (alpha * hard_loss
[CA] + (1 - alpha)
[CA] * temperature**2
[CA] * kl_loss) #FThe PyTorch listing places hard-label loss beside temperature-scaled KL divergence and returns their weighted sum. section 8.4.4
