liveBook · Manning

Cross entropy loss

Also called CrossEntropyLoss, log loss

Cross-entropy loss measures the expected negative log probability that a predicted distribution assigns to outcomes drawn from a target distribution.

How 26 Manning books explain it

Math and Architectures of Deep Learning

A classification loss that measures the mismatch between a predicted probability distribution over classes and the target distribution.

How it is used
Measures classifier-output mismatch against one-hot ground truth during training.
How deep it goes
Derives the multiclass formula, reduces it to the correct-class log probability, and explains softmax integration.
Examples
Uses cat, dog, airplane, and automobile predictions, plus height-and-weight and region-of-interest classifiers.
What you can do after
Apply it via PyTorch cross_entropy or CrossEntropyLoss to raw classifier scores and ground-truth labels.

Read how Math and Architectures of Deep Learning explains cross entropy loss →

Grokking Machine Learning

A multiclass classification loss that compares one-hot class labels with predicted class probabilities using cross-entropy.

How it is used
Measures how much probability a classifier assigns to the correct label, serving as the loss for multiclass neural-network training.
How deep it goes
Works through binary log loss, then identifies and configures its multiclass version without deriving multiclass training updates.
Examples
Uses happy-or-sad sentence predictions, a Keras classifier, and one-hot labels for handwritten digits.
What you can do after
Apply it via Keras by compiling a softmax classifier with categorical_crossentropy and monitoring loss alongside accuracy.

Read how Grokking Machine Learning explains cross entropy loss →

Inside Deep Learning: Math, Algorithms, Models

A differentiable classification loss that measures the mismatch between predicted and target probability distributions.

How it is used
Supplies the training loss for class predictions, including per-character predictions and sequence-to-sequence outputs.
How deep it goes
Explains score-to-probability conversion and correct-class confidence, then implements per-time-step loss summation without developing the statistical justification.
Examples
Uses one-hot target and predicted distributions, a character-RNN loss loop, and real-versus-fake GAN labels.
What you can do after
Apply it via `nn.CrossEntropyLoss()` to class scores and labels, or sum its calls across autoregressive time steps.

Read how Inside Deep Learning: Math, Algorithms, Models explains cross entropy loss →

Grokking Machine Learning, Second Edition

Log loss is a probabilistic classification loss that penalizes predictions according to the negative logarithm of the probability assigned to the true label.

How it is used
Measures probabilistic classification error for comparing classifiers and guiding logistic or neural-network training.
How deep it goes
Derives binary point and dataset losses, works through numerical updates, and defers the gradient-descent derivation to an appendix.
Examples
Uses happy and sad sentences, four labeled points, two competing classifiers, a logistic-update exercise, and a circular neural-network dataset.
What you can do after
Apply it via Keras categorical cross-entropy and evaluate probabilistic classifiers by summing point losses.

Read how Grokking Machine Learning, Second Edition explains cross entropy loss →

Timeless Algorithms, The Foundational Ideas

Cross-entropy loss measures the discrepancy between target outcomes and predicted probabilities.

How it is used
Uses cross entropy as a classification training objective and as a held-out measure of next-token probability quality.
How deep it goes
Derives binary cross entropy from Bernoulli likelihood and interprets it as information cost, with a numerical calculation.
Examples
Works through a pedestrian braking decision and compares held-out loss across language-model resources.
What you can do after
Evaluate and understand predicted probabilities using cross entropy while distinguishing loss from thresholded decisions and broader capabilities.

Read how Timeless Algorithms, The Foundational Ideas explains cross entropy loss →

Build AI Drug Discovery Pipelines

Cross-entropy is a loss function that measures the discrepancy between a target probability distribution and a predicted probability distribution.

How it is used
Trains drug-inhibition and antibody classifiers, and measures token reconstruction error in a molecular autoencoder.
How deep it goes
Defines binary and multiclass loss, illustrates probability penalties, and implements loss calculations for sequence reconstruction and classification.
Examples
Illustrates inhibitor probabilities, antibody class labels and reconstruction of SMILES tokens with padding ignored.
What you can do after
Applies it via PyTorch CrossEntropyLoss and F.cross_entropy to class logits or token reconstruction scores.

Read how Build AI Drug Discovery Pipelines explains cross entropy loss →

Learn Generative AI with PyTorch

A loss measure that compares predicted token distributions with the target next tokens for multiclass sequence prediction.

How it is used
Uses cross-entropy to train multicategory classifiers, including next-token music prediction, and binary cross-entropy for GAN training.
How deep it goes
Explains logits, log-softmax, target reshaping, and padding exclusion, without a mathematical derivation.
Examples
Illustrates loss calculation with ten-category clothing classification and shifted music-token sequences.
What you can do after
Apply it via PyTorch’s CrossEntropyLoss to train a next-token model while excluding padding targets.

Read how Learn Generative AI with PyTorch explains cross entropy loss →

Graph Neural Networks in Action

CrossEntropyLoss is a PyTorch loss module that measures classification error from logits and target class labels.

How it is used
Uses log loss to compare multiclass GCN and GraphSAGE predictions and validation performance.
How deep it goes
Works through loss curves and metric comparisons, but does not specify CrossEntropyLoss inputs, implementation, or a mathematical derivation.
Examples
Compares GCN and GraphSAGE validation scores, then contrasts mean, sum, max, and list aggregation results.
What you can do after
Evaluate/understand model trade-offs by reading validation-loss curves and comparing log loss with F1 score.

Read how Graph Neural Networks in Action explains cross entropy loss →

Build a Large Language Model (From Scratch)

A measure of how far a language model's predicted token distribution is from the target token distribution, used to score predictions during training and evaluation.

How it is used
Uses prediction error to evaluate and train language models, then reuses the objective for classification fine-tuning.
How deep it goes
Works through target indexing, logarithms, averaging, tensor flattening, and loader-level aggregation without deriving the loss formally.
Examples
Demonstrates shifted token batches, a 50,257-token vocabulary, manual probability scoring, and spam classification with final-token logits.
What you can do after
Apply it via PyTorch to calculate batch, training, validation, test, language-model, and classification losses.

Read how Build a Large Language Model (From Scratch) explains cross entropy loss →

Fight Fraud with Machine Learning

Cross-entropy loss measures the discrepancy between a target distribution and predicted probabilities, commonly using the negative log probability assigned to the target.

How it is used
Supports training forgery-segmentation and illicit-transaction classification models from probabilistic predictions and labels.
How deep it goes
Explains the negative-log relationship and weighted implementation, but defers a fuller mathematical derivation.
Examples
Uses forged receipt pixels, two- and three-class segmentation, and licit-versus-illicit Bitcoin transactions.
What you can do after
Apply it via PyTorch to train classifiers, adjust class weights, and monitor training and validation loss.

Read how Fight Fraud with Machine Learning explains cross entropy loss →

Deep Learning with PyTorch, Second Edition

A loss value based on cross-entropy, used here as the quantity minimized during training.

How it is used
Uses cross entropy as a training loss for image classification from class scores.
How deep it goes
Explains the equivalence to LogSoftmax plus negative log likelihood and compares gradients with mean squared error.
Examples
Uses bird-versus-airplane classification and a two-class nodule dataset to illustrate class outputs and targets.
What you can do after
Apply it via PyTorch’s nn.CrossEntropyLoss to logits and class-index targets.

Read how Deep Learning with PyTorch, Second Edition explains cross entropy loss →

Knowledge Graphs and LLMs in Action

Cross-entropy loss is a classification loss that penalizes incorrect probability assignments by comparing predicted probabilities with target labels.

How it is used
Trains graph models for link prediction and node classification by comparing predicted probabilities with relationships or labels.
How deep it goes
Explains sigmoid, negative sampling, probability penalties, and log_softmax integration, but does not derive the full optimization mathematics.
Examples
Uses an aspirin-treats-headache knowledge-graph fact and licit-versus-illicit transaction classification.
What you can do after
Apply PyTorch CrossEntropyLoss with log_softmax outputs to train a GNN classifier and evaluate label mismatches.

Read how Knowledge Graphs and LLMs in Action explains cross entropy loss →

Reinforcement Learning from Human Feedback

Cross-entropy is the expected negative log probability assigned by one distribution to outcomes drawn from another distribution.

How it is used
Trains reward models and matches language-model predictions to labeled or teacher-provided targets.
How deep it goes
Defines general and binary objectives and demonstrates masked token-level and step-level loss implementations.
Examples
Uses completion correctness labels, reasoning-step labels, and teacher next-token distributions as targets.
What you can do after
Apply it via PyTorch to masked binary token predictions and multiclass step-boundary predictions.

Read how Reinforcement Learning from Human Feedback explains cross entropy loss →

Machine Learning System Design

A classification loss that penalizes incorrect probabilistic predictions using the negative logarithm of the probability assigned to the true class.

How it is used
Uses binary cross-entropy to train search systems to predict clicks and purchases.
How deep it goes
Contrasts binary cross-entropy with focal loss for imbalanced data, without detailing the loss computation.
Examples
Uses search clicks and purchases as prediction targets, and spam versus nonspam as a classification example.
What you can do after
Apply binary cross-entropy to train click and purchase predictors, then compare results with online metrics such as CTR and conversion rate.

Read how Machine Learning System Design explains cross entropy loss →

Rearchitecting LLMs

Cross-entropy is a loss function that measures the discrepancy between a target probability distribution and a predicted probability distribution.

How it is used
Preserves accurate next-token prediction as a task objective during student-model distillation.
How deep it goes
Implements shifted logits and labels with padding exclusion, without detailing the loss derivation.
Examples
Uses “Work hard play hard” to demonstrate shifting predictions against next-token labels.
What you can do after
Apply it via PyTorch to calculate next-token loss while excluding padded labels.

Read how Rearchitecting LLMs explains cross entropy loss →

Deep Learning with Python, Third Edition

A classification loss that measures the mismatch between predicted probability distributions and true token labels.

How it is used
Guides loss selection for classification tasks with probability outputs.
How deep it goes
Defines cross-entropy as a distance between distributions and gives task-based loss choices without deriving the calculation.
Examples
Uses binary classifiers for sentiment and dogs-versus-cats classification, alongside a categorical-classification rule.
What you can do after
Apply it via Keras by selecting binary or categorical cross-entropy to match the classification task.

Read how Deep Learning with Python, Third Edition explains cross entropy loss →

How Large Language Models Work

A cross-entropy loss function measures the discrepancy between a target probability distribution and a model's predicted probability distribution.

How it is used
Replaces accuracy during training with a better-behaved proxy objective.
How deep it goes
Explains its proxy role but defers mathematical details and implementation.
Examples
Uses cancer-case counts and smooth-versus-nonsmooth function figures to illustrate the training problem.
What you can do after
Evaluate why training can optimize a proxy rather than the desired behavior.

Read how How Large Language Models Work explains cross entropy loss →

Build a Text-to-Image Generator (from Scratch)

Categorical cross-entropy loss measures the discrepancy between predicted class probabilities and the ground-truth class in multicategory classification.

How it is used
Trains multicategory classifiers, caption decoders, translation models, and text-to-image systems by minimizing prediction error.
How deep it goes
Explains target alignment and parameter minimization, then implements the loss with PyTorch without deriving its mathematical formula.
Examples
Uses image classes, the caption “a dog runs,” German–English translation batches, and discrete image-token sequences.
What you can do after
Apply it via PyTorch to train classifiers and autoregressive models against categorical or token-level targets.

Read how Build a Text-to-Image Generator (from Scratch) explains cross entropy loss →

Building Specialized AI Systems

Cross-entropy loss measures the discrepancy between target outcomes and a predicted probability distribution, penalizing low probability assigned to the correct outcome.

How it is used
Guides fine-tuning by quantifying how poorly predicted probabilities match target outcomes.
How deep it goes
Explains scalar error, probability-based penalties, loss landscapes, gradients, and the training-loop implementation.
Examples
Uses sentiment classification with a correct positive label receiving 30% probability and a basic training-loop listing.
What you can do after
Apply it via a training loop to measure prediction error and drive parameter updates during fine-tuning.

Read how Building Specialized AI Systems explains cross entropy loss →

Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI

CrossEntropyLoss is a PyTorch loss-function module that combines class-logit normalization with negative log-likelihood for classification.

How it is used
Selects the training objective for categorical prediction and helps verify that a tabular model is solving classification rather than regression.
How deep it goes
Defines the classification role and demonstrates loss-function diagnosis, but does not derive the calculation mathematically.
Examples
Uses binary Airbnb price classification, numeric-versus-string targets, MSELoss(), and CrossEntropyLoss() inspection.
What you can do after
Diagnose incorrect fastai loss selection and configure target values so a binary model uses CrossEntropyLoss().

Read how Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI explains cross entropy loss →

Natural Language Processing in Action, Second Edition

Log loss is a probabilistic prediction metric based on the negative logarithm of the probability assigned to the observed target.

How it is used
Trains a binary classifier by measuring probability predictions against observed labels.
How deep it goes
Explains the logarithmic penalty for confident errors and shows frequency weighting and a PyTorch implementation.
Examples
Uses name-based sex classification and weights examples by their dataset frequencies.
What you can do after
Apply PyTorch BCELoss with frequency weights to train a sigmoid-output binary classifier.

Read how Natural Language Processing in Action, Second Edition explains cross entropy loss →

Deep Learning with R, Third Edition

Categorical cross-entropy is a multiclass classification loss that measures the negative log-likelihood of the target class under predicted class probabilities and expects categorically encoded labels.

How it is used
Guides loss selection by matching categorical cross-entropy to many-class classification and binary cross-entropy to two-class tasks.
How deep it goes
Defines cross-entropy as a distance between distributions and gives task-selection guidance, but no categorical-specific formula or implementation.
Examples
Contrasts many-class and two-class classification as loss-selection cases, without showing a categorical-cross-entropy code example.
What you can do after
Apply it as the loss for a many-class classification task.

Read how Deep Learning with R, Third Edition explains cross entropy loss →

Build a Reasoning Model (From Scratch)

A loss function that measures the average negative log probability assigned to the correct target tokens.

How it is used
Uses answer-only token loss as the training signal and validation metric during reasoning-model distillation.
How deep it goes
Explains token alignment, answer masking, length normalization, direct computation, and multi-example evaluation without covering optimizer internals.
Examples
Demonstrates the Germany–Berlin sequence, answer-token slicing, single-example loss, and averaged train-versus-validation results.
What you can do after
Build it from scratch with shifted token tensors, answer-boundary slicing, PyTorch cross_entropy, and aggregate train or validation evaluation.

Read how Build a Reasoning Model (From Scratch) explains cross entropy loss →

Experimentation for Engineers

A loss metric that measures the discrepancy between a probability distribution predicted by a model and the observed outcome distribution.

How it is used
Uses out-of-sample cross-entropy to assess prediction quality during model improvement.
How deep it goes
Defines the metric through a production example but stops short of giving its formula or implementation.
Examples
Uses video swipe probabilities, feature-based ranking, successive model versions, and DAU outcomes.
What you can do after
Evaluate whether prediction improvements translate into the intended business metric rather than assuming correlation.

Read how Experimentation for Engineers explains cross entropy loss →

Grokking AI Algorithms, Second Edition

Cross-entropy loss is a prediction-error measure computed as the negative logarithm of the probability assigned to the correct outcome.

How it is used
Measures how wrong a language model’s next-token predictions are during training.
How deep it goes
Explains the negative-log behavior and confidence examples, then stops short of detailing full gradient derivations.
Examples
Uses probabilities of 0.99, 0.50, and 0.01, plus a token-level loss table.
What you can do after
Evaluate token predictions by translating correct-token probabilities into interpretable error values.

Read how Grokking AI Algorithms, Second Edition explains cross entropy loss →

CUDA for LLMs

Cross-entropy loss measures the discrepancy between a target probability distribution and a predicted probability distribution.

How it is used
Uses cross-entropy with output softmax for classification tasks.
How deep it goes
Derives the combined softmax–cross-entropy gradient using probabilities, one-hot labels, and batch size.
Examples
Uses a one-hot target with value 1 for the correct class and 0 for all others.
What you can do after
Build the combined output gradient from predicted probabilities, one-hot labels, and batch size.

Read how CUDA for LLMs explains cross entropy loss →

Also covered in

What to know first

Related concepts

Questions about cross entropy loss

What is cross entropy loss?

Cross-entropy loss measures the expected negative log probability that a predicted distribution assigns to outcomes drawn from a target distribution.

Which Manning books explain cross entropy loss?

Math and Architectures of Deep Learning, Grokking Machine Learning, Inside Deep Learning: Math, Algorithms, Models, Grokking Machine Learning, Second Edition, Timeless Algorithms, The Foundational Ideas, Build AI Drug Discovery Pipelines, Learn Generative AI with PyTorch, Graph Neural Networks in Action and 18 more.

What should I know before learning cross entropy loss?

Manning books that teach cross entropy loss usually introduce objective function, language model, classifier and probability distribution first.

What kinds of cross entropy loss are there?

Kinds of cross entropy loss include focal loss.

What is cross entropy loss a kind of?

Cross entropy loss is a kind of objective function. An objective function is a mathematical function that assigns values to candidate solutions or parameter settings, which an optimization procedure seeks to minimize or maximize.

What is cross entropy loss also called?

Cross entropy loss is also called CrossEntropyLoss or log loss.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.