Causal language models predict the next token at each position. To calculate task loss, compare each logit position with the following label: drop the final logit, which has no next token, and the first label, which has no preceding prediction. Mark padding labels with -100 so cross-entropy ignores them. section 6.3.1
Aligning a sequence
For “Work hard play hard,” compare the logits for “Work,” “hard,” and “play” with the labels “hard,” “play,” and “hard”; the final logit has no target. section 6.3.1
shift_logits = student_logits[..., :-1, :].contiguous() #A
shift_labels = labels[..., 1:].contiguous() #B
loss_task = F.cross_entropy(
shift_logits.view(-1, shift_logits.size(-1)), #C
shift_labels.view(-1),
ignore_index=-100 #D
)
The PyTorch listing slices logits and labels into next-token pairs before flattening them for cross-entropy. section 6.3.1
