liveBook · Manning

Cross entropy loss

As explained in Grokking Machine Learning, Second Edition

Log loss is a probabilistic classification loss that penalizes predictions according to the negative logarithm of the probability assigned to the true label.

Cross entropy loss, also called log loss, measures how well a classifier’s probabilities match the true labels. For each point, it looks at the probability assigned to that point’s actual label, takes its natural logarithm, and changes the sign. A confident correct prediction therefore gets a small loss; a confident wrong prediction gets a large one. section 6.14 section 6.4

Table 6.5 Calculation of the log loss for the points in our dataset. Notice that points that are well classified (2 and 4) have a small log loss, whereas points that are poorly classified (1 and 3) have a large log loss.
Point
Label
Predicted label
Probability of being its label
Log loss
1
0 (Sad)
0.953
0.047
–ln(0.047) = 3.049
2
1 (Happy)
0.731
0.731
–ln(0.731) = 0.313
3
1 (Happy)
0.119
0.119
–ln(0.119) = 2.127
4
0 (Sad)
0.119
0.881
–ln(0.881) = 0.127

These four labeled points make the pattern concrete: probability 0.731 for the true label gives loss 0.313, while probability 0.047 gives loss 3.049. section 6.4

For a binary label y and predicted probability ŷ of label 1, the loss is −y ln(ŷ) − (1−y) ln(1−ŷ). When y is 1, only the first part matters: loss is −ln(ŷ). When y is 0, it becomes −ln(1−ŷ). The total loss for a dataset is the sum of its point losses. section 6.5 section 6.14

Comparing two classifiers

On a four-sentence happy/sad dataset, one classifier has total log loss 5.616 and another has 1.067. The lower value agrees with the observed predictions: the second classifier places the happy sentences in the positive zone and the sad sentences in the negative zone. section 6.5

Log loss is especially useful during training because it responds even when a point is already classified correctly. The logistic update uses the difference y−ŷ to adjust weights and bias, moving the predicted probability closer to the label and reducing that point’s log loss. Repeating updates while watching total loss trains a logistic classifier. section 6.6 section 6.7

The same idea appears in two common classification settings:

  • Binary logistic classification uses log loss with probabilities between 0 and 1. section 6.14
  • Multi-class neural-network classification uses categorical cross-entropy, the multivariate version of log loss, when labels have more than one column. section 10.18

Small Keras use

For a two-class circular dataset, labels are converted to two columns, then a neural network with a softmax output is compiled with loss='categorical_crossentropy'. Over 100 epochs with batches of 10, the reported loss falls from 0.5473 to 0.2110 while accuracy rises from 0.7182 to 0.9000. section 10.18

How Grokking Machine Learning, Second Edition covers it

How deep it goes
Derives binary point and dataset losses, works through numerical updates, and defers the gradient-descent derivation to an appendix.
Examples
Uses happy and sad sentences, four labeled points, two competing classifiers, a logistic-update exercise, and a circular neural-network dataset.
What it assumes
Introduces loss from binary labels and predicted probabilities, while using sigmoid predictions and natural logarithms.
What you can do after
Apply it via Keras categorical cross-entropy and evaluate probabilistic classifiers by summing point losses.
How it connects
Connects logistic classifiers, sigmoid activation, gradient descent, softmax, categorical cross-entropy, and neural networks.

Also explained in

Compare how 26 books explain cross entropy loss →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning, Second Edition. The section links point to the parts of the book it draws on.