A hard label identifies the dataset-designated correct class or token. Unlike a soft label, it gives no probability distribution over alternatives. section 6.3 section 6.3.1
In knowledge distillation, hard labels provide the task objective: they encourage the student to predict the dataset’s correct answer. Combined with soft-label logits loss, they balance task accuracy against learning the teacher’s confidence across alternatives. section 6.3 section 6.5.1
Next-token targets
For “Work hard play hard,” the model’s prediction after “Work” is compared with “hard,” and each later prediction is aligned with the following token. The first label and final logit are omitted; padding labels marked -100 are excluded from cross-entropy. section 6.3.1
shift_logits = student_logits[..., :-1, :].contiguous() #A
shift_labels = labels[..., 1:].contiguous() #B
loss_task = F.cross_entropy(
shift_logits.view(-1, shift_logits.size(-1)), #C
shift_labels.view(-1),
ignore_index=-100 #D
)
Inspect the PyTorch snippet for shifting logits and labels before cross-entropy, with padded labels ignored. section 6.3.1
