For a classifier, the loss needs to say how well its output matches the correct class. Cross-entropy loss does this by combining softmax with negative log likelihood: softmax turns class scores into probabilities, and negative log likelihood penalizes the probability assigned to the correct class. section 7.5
In PyTorch, pass logits—the class scores before softmax—directly to cross-entropy loss. This gives the same result as passing the output of nn.LogSoftmax to nn.NLLLoss. Do not add a softmax output layer before using the direct-logits approach. section 7.2.4
Choosing a loss for two classes
A bird-versus-airplane network produces two class scores. Those scores can go directly to cross-entropy loss; alternatively, a network ending in nn.LogSoftmax can use nn.NLLLoss with a class index of 0 or 1. In a separate nodule-versus-non-nodule dataset, the classification tensor has two elements, one for each candidate class, because the classifier provides one output per class. section 7.2.4 section 12.5
Figure 7.11 The cross-entropy (left) and MSE between predicted probabilities and the target probability vector (right) as functions of the predicted scores—that is, before the (log-) softmax.
Compare the loss curves against mean squared error (MSE). Cross-entropy retains a useful slope even for nearly correct predictions; MSE can become too flat, including when predictions are substantially wrong. section 7.2.4
