A classifier can make the right decision while assigning only modest probability to the correct outcome. Cross entropy loss measures the discrepancy between predicted probabilities and observed outcomes, retaining information that a thresholded decision discards. section 11.3.3 section 10.3.3
For binary classification, y is the observed class, 0 or 1, and ŷ is the predicted probability of class 1. When y = 1, the loss is −log(ŷ); when y = 0, it is −log(1 − ŷ). section 11.3.3
The loss is the negative logarithm of the probability assigned to the observed outcome. It approaches zero when that probability approaches one and grows large when it approaches zero. Confident mistakes are therefore penalized more heavily than uncertain ones. Base-2 logarithms express the information cost in bits. section 11.4.2 section 11.2.3 section 5.4.1
A pedestrian prediction
Suppose a pedestrian is present and a network assigns that outcome probability 0.68. With a braking threshold of 0.30, the decision is correct, but the loss is still −log(0.68) ≈ 0.39. A probability of 0.99 would produce the same decision but a much smaller loss, about 0.01. section 11.3.3 section 11.4.2
For independent binary observations, the likelihood is the product of the probabilities assigned to their observed classes. Taking its negative logarithm gives the sum of their cross entropy losses. Minimizing that sum is therefore equivalent to maximizing likelihood under the model. Backpropagation calculates how parameters influenced the loss, and gradient descent uses the resulting gradients to update them. section 11.4.2 section 11.8
The same quantity links learning to information theory: cross entropy is the average code length obtained when one probability distribution represents outcomes generated by another. An inaccurate probability model can require longer codes; in classification, assigning low probability to observed outcomes produces high loss. section 10.3.3
In next-token prediction, cross entropy evaluates probabilities assigned to observed tokens. Held-out loss can test whether improvements extend beyond training data, but it does not directly measure reasoning, factual accuracy, or reliability. Likewise, reducing training loss alone does not guarantee generalization. section 12.3.1 section 13.1.2 section 13.3 section 10.5.1

























