Cross-entropy loss compares a model’s next-token predictions with hard labels from the dataset. In distillation, this task loss helps the student retain accurate predictions while also learning from the teacher. section 6.3.1
shift_logits = student_logits[..., :-1, :].contiguous() #A
shift_labels = labels[..., 1:].contiguous() #B
loss_task = F.cross_entropy(
shift_logits.view(-1, shift_logits.size(-1)), #C
shift_labels.view(-1),
ignore_index=-100 #D
)
Shift logits and labels so each prediction is compared with the next token; ignore padded labels marked -100. section 6.3.1
Next-token alignment
For “Work hard play hard,” compare the logits at “Work” with the label “hard,” then align each later position with its following token. section 6.3.1

























