Cross entropy loss, also called log loss, measures how well a classifier’s probabilities match the true labels. For each point, it looks at the probability assigned to that point’s actual label, takes its natural logarithm, and changes the sign. A confident correct prediction therefore gets a small loss; a confident wrong prediction gets a large one. section 6.14 section 6.4
Table 6.5 Calculation of the log loss for the points in our dataset. Notice that points that are well classified (2 and 4) have a small log loss, whereas points that are poorly classified (1 and 3) have a large log loss.
|
Point
|
Label
|
Predicted label
|
Probability of being its label
|
Log loss
|
|
1
|
0 (Sad)
|
0.953
|
0.047
|
–ln(0.047) = 3.049
|
|
2
|
1 (Happy)
|
0.731
|
0.731
|
–ln(0.731) = 0.313
|
|
3
|
1 (Happy)
|
0.119
|
0.119
|
–ln(0.119) = 2.127
|
|
4
|
0 (Sad)
|
0.119
|
0.881
|
–ln(0.881) = 0.127
|
These four labeled points make the pattern concrete: probability 0.731 for the true label gives loss 0.313, while probability 0.047 gives loss 3.049. section 6.4
For a binary label y and predicted probability ŷ of label 1, the loss is −y ln(ŷ) − (1−y) ln(1−ŷ). When y is 1, only the first part matters: loss is −ln(ŷ). When y is 0, it becomes −ln(1−ŷ). The total loss for a dataset is the sum of its point losses. section 6.5 section 6.14
Comparing two classifiers
On a four-sentence happy/sad dataset, one classifier has total log loss 5.616 and another has 1.067. The lower value agrees with the observed predictions: the second classifier places the happy sentences in the positive zone and the sad sentences in the negative zone. section 6.5
Log loss is especially useful during training because it responds even when a point is already classified correctly. The logistic update uses the difference y−ŷ to adjust weights and bias, moving the predicted probability closer to the label and reducing that point’s log loss. Repeating updates while watching total loss trains a logistic classifier. section 6.6 section 6.7
The same idea appears in two common classification settings:
- Binary logistic classification uses log loss with probabilities between 0 and 1. section 6.14
- Multi-class neural-network classification uses categorical cross-entropy, the multivariate version of log loss, when labels have more than one column. section 10.18
Small Keras use
For a two-class circular dataset, labels are converted to two columns, then a neural network with a softmax output is compiled with loss='categorical_crossentropy'. Over 100 epochs with batches of 10, the reported loss falls from 0.5473 to 0.2110 while accuracy rises from 0.7182 to 0.9000. section 10.18

























