Cross-entropy loss measures how confidently a language model predicts each correct next token. It averages the negative log probabilities over target answer tokens, making sequence losses comparable across lengths. section 8.6
Compute the answer-only loss in four implementation steps:
- Shift the token sequence into model inputs and next-token targets. section 8.6
- Run the model and retain logits beginning at the answer boundary, excluding prompt predictions. section 8.6
- Apply cross_entropy to answer logits and reference answer tokens; values closer to zero indicate greater confidence. section 8.6
- Average losses across examples for training and validation monitoring, restoring training mode after evaluation. section 8.6
Direct answer-token calculation
For “The capital of Germany is Berlin,” five target predictions produce a summed log probability of −16.6250; negating and averaging gives cross-entropy 3.325. A PyTorch implementation slices answer logits and targets, then calls torch.nn.functional.cross_entropy. section 8.6 section 8.6
Figure 8.12 Illustration of token and sequence log probabilities. The log probabilities of the correct next tokens are summed to obtain the sequence log probability, which is the basis for the cross-entropy loss used later in this chapter.
Trace how token log probabilities combine into a sequence log probability and then become the cross-entropy training signal. section 8.6
