Suppose a classifier must choose which handwritten digit appears in an image. A correct guess is useful, but so is knowing how much probability the classifier gave the right digit. Cross-entropy loss measures that difference: it is small when the true class receives high probability and large when it receives low probability. The multiclass form is called categorical cross-entropy. section 6.6 section 10.3
Start with two classes. If y is the label, either 0 or 1, and ŷ is the predicted probability of class 1, the loss for one point is −y ln(ŷ) − (1 − y) ln(1 − ŷ). In plain language, take the probability assigned to the actual label, then take its negative natural logarithm. Add the losses across points to get the dataset’s total log loss. section 6.1 section 6.6
Table 6.5 Calculation of the log loss for the points in our dataset. Notice that points that are well classified (2 and 4) have a small log loss, whereas points that are poorly classified (1 and 3) have a large log loss. (view table figure)
| Point |
Label |
Predicted label |
Probability of being its label |
Log loss |
| 1 |
0 (Sad) |
0.953 |
0.047 |
–ln(0.047) = 3.049 |
| 2 |
1 (Happy) |
0.731 |
0.731 |
–ln(0.731) = 0.313 |
| 3 |
1 (Happy) |
0.119 |
0.119 |
–ln(0.119) = 2.127 |
| 4 |
0 (Sad) |
0.119 |
0.881 |
–ln(0.881) = 0.127 |
Compare the four labels, predicted probabilities, and losses. A point whose true label receives probability 0.047 has loss 3.049; one whose true label receives probability 0.881 has loss 0.127. section 6.1
For several classes, represent the true label as a one-hot vector: its class has entry 1 and every other class has entry 0. A softmax output gives a predicted probability for each class. Categorical cross-entropy is the multivariate version of log loss; equivalently, for one example it is −Σᵢ yᵢ ln(pᵢ). Because only one yᵢ is 1, the calculation selects the predicted probability of the true class. section 6.6 section 10.3
Figure 10.24 Before training the neural network, we preprocess the images and the labels in the following way. We shape the rectangular image into a long vector by concatenating the rows. We then convert each label into a vector of length 10 with only one non-zero entry in the position of the corresponding label.
See how a digit label becomes a length-10 vector with a single nonzero entry—the label format used for multiclass training. section 10.3
To use it in Keras, pair a softmax output with one-hot labels and compile the model with loss='categorical_crossentropy'. Accuracy can be reported separately: loss responds to the probabilities, while accuracy reports classification results. Try training the same model with different optimizers and compare its reported loss and accuracy. section 10.3
