Logit loss helps a student recover behavior from a teacher during distillation. Rather than learning only the dataset’s correct next token, the student also learns to match the teacher’s probability distribution across the vocabulary. section 6.3.1 section 6.6
For next-token distillation, calculate the loss in stages:
- Divide both models’ logits by the same temperature. Apply log-softmax to the student logits and softmax to the teacher logits. section 6.3.1
- Drop the final logit position from both outputs, then flatten the remaining positions while retaining the vocabulary dimension. section 6.3.1
- Calculate KL divergence using the student log-probabilities and teacher probabilities, with batchmean reduction. Multiply the result by temperature squared. section 6.3.1
student_soft = F.log_softmax(
student_logits / temperature,
dim=-1) #A
teacher_soft = F.softmax(
teacher_logits / temperature,
dim=-1) #B
student_soft = student_soft[..., :-1, :].contiguous()
teacher_soft = teacher_soft[..., :-1, :].contiguous()
loss_logits = F.kl_div(
student_soft.view(-1, student_soft.size(-1)),
teacher_soft.view(-1, teacher_soft.size(-1)),
reduction='batchmean' #C
) * (temperature ** 2) #D
The listing puts temperature scaling, position slicing, reshaping, and KL divergence together. section 6.3.1
Logit loss can be combined with task loss and hidden loss. One baseline weights them 0.4, 0.4, and 0.2 respectively. Adjusting the weights changes the emphasis: generative models prioritize output behavior, while representation-focused models may need more hidden-state alignment. section 6.3.1 section 6.8
Hard labels are not always necessary. Reported large-scale comparisons found that distillation retraining outperformed standard fine-tuning; Mistral also reported that a pure logits objective outperformed its tested combinations with next-token prediction loss. For a specific task, retaining hard-label loss can instead emphasize task results while logits help stabilize learning and generalization. section 6.6
