liveBook · Manning

Cross entropy loss

As explained in Timeless Algorithms, The Foundational Ideas

Cross-entropy loss measures the discrepancy between target outcomes and predicted probabilities.

A classifier can make the right decision while assigning only modest probability to the correct outcome. Cross entropy loss measures the discrepancy between predicted probabilities and observed outcomes, retaining information that a thresholded decision discards. section 11.3.3 section 10.3.3

L(y,y^)=-[ylog⁡(y^)+(1-y)log⁡(1-y^)]

For binary classification, y is the observed class, 0 or 1, and ŷ is the predicted probability of class 1. When y = 1, the loss is −log(ŷ); when y = 0, it is −log(1 − ŷ). section 11.3.3

The loss is the negative logarithm of the probability assigned to the observed outcome. It approaches zero when that probability approaches one and grows large when it approaches zero. Confident mistakes are therefore penalized more heavily than uncertain ones. Base-2 logarithms express the information cost in bits. section 11.4.2 section 11.2.3 section 5.4.1

A pedestrian prediction

Suppose a pedestrian is present and a network assigns that outcome probability 0.68. With a braking threshold of 0.30, the decision is correct, but the loss is still −log(0.68) ≈ 0.39. A probability of 0.99 would produce the same decision but a much smaller loss, about 0.01. section 11.3.3 section 11.4.2

For independent binary observations, the likelihood is the product of the probabilities assigned to their observed classes. Taking its negative logarithm gives the sum of their cross entropy losses. Minimizing that sum is therefore equivalent to maximizing likelihood under the model. Backpropagation calculates how parameters influenced the loss, and gradient descent uses the resulting gradients to update them. section 11.4.2 section 11.8

The same quantity links learning to information theory: cross entropy is the average code length obtained when one probability distribution represents outcomes generated by another. An inaccurate probability model can require longer codes; in classification, assigning low probability to observed outcomes produces high loss. section 10.3.3

In next-token prediction, cross entropy evaluates probabilities assigned to observed tokens. Held-out loss can test whether improvements extend beyond training data, but it does not directly measure reasoning, factual accuracy, or reliability. Likewise, reducing training loss alone does not guarantee generalization. section 12.3.1 section 13.1.2 section 13.3 section 10.5.1

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Derives binary cross entropy from Bernoulli likelihood and interprets it as information cost, with a numerical calculation.
Examples
Works through a pedestrian braking decision and compares held-out loss across language-model resources.
What you can do after
Evaluate and understand predicted probabilities using cross entropy while distinguishing loss from thresholded decisions and broader capabilities.
How it connects
Connects cross entropy to information theory, maximum likelihood, backpropagation, gradient descent, and generalization.

Also explained in

Compare how 26 books explain cross entropy loss →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.