Cross-entropy loss measures the discrepancy between predicted class probabilities and the ground-truth class. It supports multicategory image classification and sequence prediction by penalizing incorrect predictions and minimizing the loss during training. section 3.4.1 section 4.5
Apply cross-entropy loss through a supervised prediction loop:
- Prepare ground-truth targets, such as class labels or a target token sequence. section 3.4.1 section 4.5
- Compare model outputs with those targets, aligning each prediction with the correct class or next token. section 4.5 section 2.2.2
- Modify model parameters to minimize the resulting loss during training. section 4.5 section 12.1.1
loss_fn = nn.CrossEntropyLoss()
Instantiate categorical cross-entropy directly with PyTorch’s CrossEntropyLoss implementation. section 3.4.1
Caption and image-token prediction
For the caption “a dog runs,” the decoder receives “<start> a dog runs” and learns to predict “a dog runs <end>.” In text-to-image generation, the same objective compares predicted image-token sequences with actual sequences. section 4.5 section 12.1.1
