liveBook · Manning

Cross entropy loss

As explained in Rearchitecting LLMs

Cross-entropy is a loss function that measures the discrepancy between a target probability distribution and a predicted probability distribution.

Cross-entropy loss compares a model’s next-token predictions with hard labels from the dataset. In distillation, this task loss helps the student retain accurate predictions while also learning from the teacher. section 6.3.1

shift_logits = student_logits[..., :-1, :].contiguous() #A
shift_labels = labels[..., 1:].contiguous() #B
loss_task = F.cross_entropy(
    shift_logits.view(-1, shift_logits.size(-1)), #C
    shift_labels.view(-1),
    ignore_index=-100 #D
)

Shift logits and labels so each prediction is compared with the next token; ignore padded labels marked -100. section 6.3.1

Next-token alignment

For “Work hard play hard,” compare the logits at “Work” with the label “hard,” then align each later position with its following token. section 6.3.1

How Rearchitecting LLMs covers it

How deep it goes
Implements shifted logits and labels with padding exclusion, without detailing the loss derivation.
Examples
Uses “Work hard play hard” to demonstrate shifting predictions against next-token labels.
What it assumes
Assumes tokenized sequences and model logits, while explaining next-token label alignment.
What you can do after
Apply it via PyTorch to calculate next-token loss while excluding padded labels.
How it connects
Connects cross-entropy to knowledge distillation and next-token prediction.

Also explained in

Compare how 26 books explain cross entropy loss →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.