Cross entropy loss trains a classifier by measuring the mismatch between its predicted class distribution and the ground-truth distribution. For a multiclass example, the target is usually one-hot: one correct class has probability 1 and every other class has probability 0. The model should emit probabilities that sum to 1. section 6.3 section 9.1.3
For target vector ȳ and predicted probabilities y, the loss is −Σj ȳj log(yj). One-hot targets make every term vanish except the correct-class term, so the loss becomes −log(yj*), where j* is the true class. Assigning probability 1 to that class gives loss 0; assigning it little probability produces a large loss. section 9.1.3 section 9.4
Good and bad class probabilities
For a cat target [1, 0, 0, 0], [0.8, 0.15, 0.04, 0.01] is a good prediction because it concentrates probability on cat. A uniform prediction [0.25, 0.25, 0.25, 0.25] is worse: it treats every class as equally likely. Their reported losses are 0.22 and 1.38, respectively. section 6.3
Cross entropy includes an offset: when target and prediction distributions are identical, it equals the target entropy rather than zero. Subtracting that fixed target entropy yields Kullback–Leibler divergence. Because the target distribution is fixed during classifier training, minimizing cross entropy is equivalent to minimizing KL divergence from target to prediction. section 6.4
Classifier networks commonly produce raw, unbounded class scores called logits. Softmax converts them to probabilities, but PyTorch combines softmax and cross entropy in one operation for better numerical behavior. Supply raw scores and class labels to that combined loss during training; use softmax when turning scores into prediction probabilities. section 9.1.6 section 9.2.3
Listing 9.5 PyTorch code for softmax cross-entropy loss
from torch.nn.functional import cross_entropy scores = torch.tensor([[9.99, 10, 0.01, -10]]) y_gt = torch.tensor([1]) #1 loss = cross_entropy(scores, y_gt) #2
The PyTorch example shows the combined cross_entropy operation receiving logits and an integer ground-truth class index. section 9.1.6
In minibatch training, compute cross entropy for sampled inputs and use its gradients to update parameters with stochastic gradient descent. The same classification term can also be paired with a regression loss in object detection, where Fast R-CNN classifies regions while regressing bounding-box offsets. section 9.4 section 11.4.3
