Cross entropy loss
Cross-entropy loss measures the expected negative log probability that a predicted distribution assigns to outcomes drawn from a target distribution.
Math and Architectures of Deep Learning
A classification loss that measures the mismatch between a predicted probability distribution over classes and the target distribution.
- How it is used
- Measures classifier-output mismatch against one-hot ground truth during training.
- How deep it goes
- Derives the multiclass formula, reduces it to the correct-class log probability, and explains softmax integration.
- Examples
- Uses cat, dog, airplane, and automobile predictions, plus height-and-weight and region-of-interest classifiers.
- What you can do after
- Apply it via PyTorch cross_entropy or CrossEntropyLoss to raw classifier scores and ground-truth labels.
Read how Math and Architectures of Deep Learning explains cross entropy loss →
Grokking Machine Learning
A multiclass classification loss that compares one-hot class labels with predicted class probabilities using cross-entropy.
- How it is used
- Measures how much probability a classifier assigns to the correct label, serving as the loss for multiclass neural-network training.
- How deep it goes
- Works through binary log loss, then identifies and configures its multiclass version without deriving multiclass training updates.
- Examples
- Uses happy-or-sad sentence predictions, a Keras classifier, and one-hot labels for handwritten digits.
- What you can do after
- Apply it via Keras by compiling a softmax classifier with categorical_crossentropy and monitoring loss alongside accuracy.
Read how Grokking Machine Learning explains cross entropy loss →
Inside Deep Learning: Math, Algorithms, Models
A differentiable classification loss that measures the mismatch between predicted and target probability distributions.
- How it is used
- Supplies the training loss for class predictions, including per-character predictions and sequence-to-sequence outputs.
- How deep it goes
- Explains score-to-probability conversion and correct-class confidence, then implements per-time-step loss summation without developing the statistical justification.
- Examples
- Uses one-hot target and predicted distributions, a character-RNN loss loop, and real-versus-fake GAN labels.
- What you can do after
- Apply it via `nn.CrossEntropyLoss()` to class scores and labels, or sum its calls across autoregressive time steps.
Read how Inside Deep Learning: Math, Algorithms, Models explains cross entropy loss →
Grokking Machine Learning, Second Edition
Log loss is a probabilistic classification loss that penalizes predictions according to the negative logarithm of the probability assigned to the true label.
- How it is used
- Measures probabilistic classification error for comparing classifiers and guiding logistic or neural-network training.
- How deep it goes
- Derives binary point and dataset losses, works through numerical updates, and defers the gradient-descent derivation to an appendix.
- Examples
- Uses happy and sad sentences, four labeled points, two competing classifiers, a logistic-update exercise, and a circular neural-network dataset.
- What you can do after
- Apply it via Keras categorical cross-entropy and evaluate probabilistic classifiers by summing point losses.
Read how Grokking Machine Learning, Second Edition explains cross entropy loss →
Timeless Algorithms, The Foundational Ideas
Cross-entropy loss measures the discrepancy between target outcomes and predicted probabilities.
- How it is used
- Uses cross entropy as a classification training objective and as a held-out measure of next-token probability quality.
- How deep it goes
- Derives binary cross entropy from Bernoulli likelihood and interprets it as information cost, with a numerical calculation.
- Examples
- Works through a pedestrian braking decision and compares held-out loss across language-model resources.
- What you can do after
- Evaluate and understand predicted probabilities using cross entropy while distinguishing loss from thresholded decisions and broader capabilities.
Read how Timeless Algorithms, The Foundational Ideas explains cross entropy loss →
Build AI Drug Discovery Pipelines
Cross-entropy is a loss function that measures the discrepancy between a target probability distribution and a predicted probability distribution.
- How it is used
- Trains drug-inhibition and antibody classifiers, and measures token reconstruction error in a molecular autoencoder.
- How deep it goes
- Defines binary and multiclass loss, illustrates probability penalties, and implements loss calculations for sequence reconstruction and classification.
- Examples
- Illustrates inhibitor probabilities, antibody class labels and reconstruction of SMILES tokens with padding ignored.
- What you can do after
- Applies it via PyTorch CrossEntropyLoss and F.cross_entropy to class logits or token reconstruction scores.
Read how Build AI Drug Discovery Pipelines explains cross entropy loss →
Learn Generative AI with PyTorch
A loss measure that compares predicted token distributions with the target next tokens for multiclass sequence prediction.
- How it is used
- Uses cross-entropy to train multicategory classifiers, including next-token music prediction, and binary cross-entropy for GAN training.
- How deep it goes
- Explains logits, log-softmax, target reshaping, and padding exclusion, without a mathematical derivation.
- Examples
- Illustrates loss calculation with ten-category clothing classification and shifted music-token sequences.
- What you can do after
- Apply it via PyTorch’s CrossEntropyLoss to train a next-token model while excluding padding targets.
Read how Learn Generative AI with PyTorch explains cross entropy loss →
Graph Neural Networks in Action
CrossEntropyLoss is a PyTorch loss module that measures classification error from logits and target class labels.
- How it is used
- Uses log loss to compare multiclass GCN and GraphSAGE predictions and validation performance.
- How deep it goes
- Works through loss curves and metric comparisons, but does not specify CrossEntropyLoss inputs, implementation, or a mathematical derivation.
- Examples
- Compares GCN and GraphSAGE validation scores, then contrasts mean, sum, max, and list aggregation results.
- What you can do after
- Evaluate/understand model trade-offs by reading validation-loss curves and comparing log loss with F1 score.
Read how Graph Neural Networks in Action explains cross entropy loss →
Build a Large Language Model (From Scratch)
A measure of how far a language model's predicted token distribution is from the target token distribution, used to score predictions during training and evaluation.
- How it is used
- Uses prediction error to evaluate and train language models, then reuses the objective for classification fine-tuning.
- How deep it goes
- Works through target indexing, logarithms, averaging, tensor flattening, and loader-level aggregation without deriving the loss formally.
- Examples
- Demonstrates shifted token batches, a 50,257-token vocabulary, manual probability scoring, and spam classification with final-token logits.
- What you can do after
- Apply it via PyTorch to calculate batch, training, validation, test, language-model, and classification losses.
Read how Build a Large Language Model (From Scratch) explains cross entropy loss →
Fight Fraud with Machine Learning
Cross-entropy loss measures the discrepancy between a target distribution and predicted probabilities, commonly using the negative log probability assigned to the target.
- How it is used
- Supports training forgery-segmentation and illicit-transaction classification models from probabilistic predictions and labels.
- How deep it goes
- Explains the negative-log relationship and weighted implementation, but defers a fuller mathematical derivation.
- Examples
- Uses forged receipt pixels, two- and three-class segmentation, and licit-versus-illicit Bitcoin transactions.
- What you can do after
- Apply it via PyTorch to train classifiers, adjust class weights, and monitor training and validation loss.
Read how Fight Fraud with Machine Learning explains cross entropy loss →
Deep Learning with PyTorch, Second Edition
A loss value based on cross-entropy, used here as the quantity minimized during training.
- How it is used
- Uses cross entropy as a training loss for image classification from class scores.
- How deep it goes
- Explains the equivalence to LogSoftmax plus negative log likelihood and compares gradients with mean squared error.
- Examples
- Uses bird-versus-airplane classification and a two-class nodule dataset to illustrate class outputs and targets.
- What you can do after
- Apply it via PyTorch’s nn.CrossEntropyLoss to logits and class-index targets.
Read how Deep Learning with PyTorch, Second Edition explains cross entropy loss →
Knowledge Graphs and LLMs in Action
Cross-entropy loss is a classification loss that penalizes incorrect probability assignments by comparing predicted probabilities with target labels.
- How it is used
- Trains graph models for link prediction and node classification by comparing predicted probabilities with relationships or labels.
- How deep it goes
- Explains sigmoid, negative sampling, probability penalties, and log_softmax integration, but does not derive the full optimization mathematics.
- Examples
- Uses an aspirin-treats-headache knowledge-graph fact and licit-versus-illicit transaction classification.
- What you can do after
- Apply PyTorch CrossEntropyLoss with log_softmax outputs to train a GNN classifier and evaluate label mismatches.
Read how Knowledge Graphs and LLMs in Action explains cross entropy loss →
Reinforcement Learning from Human Feedback
Cross-entropy is the expected negative log probability assigned by one distribution to outcomes drawn from another distribution.
- How it is used
- Trains reward models and matches language-model predictions to labeled or teacher-provided targets.
- How deep it goes
- Defines general and binary objectives and demonstrates masked token-level and step-level loss implementations.
- Examples
- Uses completion correctness labels, reasoning-step labels, and teacher next-token distributions as targets.
- What you can do after
- Apply it via PyTorch to masked binary token predictions and multiclass step-boundary predictions.
Read how Reinforcement Learning from Human Feedback explains cross entropy loss →
Machine Learning System Design
A classification loss that penalizes incorrect probabilistic predictions using the negative logarithm of the probability assigned to the true class.
- How it is used
- Uses binary cross-entropy to train search systems to predict clicks and purchases.
- How deep it goes
- Contrasts binary cross-entropy with focal loss for imbalanced data, without detailing the loss computation.
- Examples
- Uses search clicks and purchases as prediction targets, and spam versus nonspam as a classification example.
- What you can do after
- Apply binary cross-entropy to train click and purchase predictors, then compare results with online metrics such as CTR and conversion rate.
Read how Machine Learning System Design explains cross entropy loss →
Rearchitecting LLMs
Cross-entropy is a loss function that measures the discrepancy between a target probability distribution and a predicted probability distribution.
- How it is used
- Preserves accurate next-token prediction as a task objective during student-model distillation.
- How deep it goes
- Implements shifted logits and labels with padding exclusion, without detailing the loss derivation.
- Examples
- Uses “Work hard play hard” to demonstrate shifting predictions against next-token labels.
- What you can do after
- Apply it via PyTorch to calculate next-token loss while excluding padded labels.
Deep Learning with Python, Third Edition
A classification loss that measures the mismatch between predicted probability distributions and true token labels.
- How it is used
- Guides loss selection for classification tasks with probability outputs.
- How deep it goes
- Defines cross-entropy as a distance between distributions and gives task-based loss choices without deriving the calculation.
- Examples
- Uses binary classifiers for sentiment and dogs-versus-cats classification, alongside a categorical-classification rule.
- What you can do after
- Apply it via Keras by selecting binary or categorical cross-entropy to match the classification task.
Read how Deep Learning with Python, Third Edition explains cross entropy loss →
How Large Language Models Work
A cross-entropy loss function measures the discrepancy between a target probability distribution and a model's predicted probability distribution.
- How it is used
- Replaces accuracy during training with a better-behaved proxy objective.
- How deep it goes
- Explains its proxy role but defers mathematical details and implementation.
- Examples
- Uses cancer-case counts and smooth-versus-nonsmooth function figures to illustrate the training problem.
- What you can do after
- Evaluate why training can optimize a proxy rather than the desired behavior.
Read how How Large Language Models Work explains cross entropy loss →
Build a Text-to-Image Generator (from Scratch)
Categorical cross-entropy loss measures the discrepancy between predicted class probabilities and the ground-truth class in multicategory classification.
- How it is used
- Trains multicategory classifiers, caption decoders, translation models, and text-to-image systems by minimizing prediction error.
- How deep it goes
- Explains target alignment and parameter minimization, then implements the loss with PyTorch without deriving its mathematical formula.
- Examples
- Uses image classes, the caption “a dog runs,” German–English translation batches, and discrete image-token sequences.
- What you can do after
- Apply it via PyTorch to train classifiers and autoregressive models against categorical or token-level targets.
Read how Build a Text-to-Image Generator (from Scratch) explains cross entropy loss →
Building Specialized AI Systems
Cross-entropy loss measures the discrepancy between target outcomes and a predicted probability distribution, penalizing low probability assigned to the correct outcome.
- How it is used
- Guides fine-tuning by quantifying how poorly predicted probabilities match target outcomes.
- How deep it goes
- Explains scalar error, probability-based penalties, loss landscapes, gradients, and the training-loop implementation.
- Examples
- Uses sentiment classification with a correct positive label receiving 30% probability and a basic training-loop listing.
- What you can do after
- Apply it via a training loop to measure prediction error and drive parameter updates during fine-tuning.
Read how Building Specialized AI Systems explains cross entropy loss →
Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI
CrossEntropyLoss is a PyTorch loss-function module that combines class-logit normalization with negative log-likelihood for classification.
- How it is used
- Selects the training objective for categorical prediction and helps verify that a tabular model is solving classification rather than regression.
- How deep it goes
- Defines the classification role and demonstrates loss-function diagnosis, but does not derive the calculation mathematically.
- Examples
- Uses binary Airbnb price classification, numeric-versus-string targets, MSELoss(), and CrossEntropyLoss() inspection.
- What you can do after
- Diagnose incorrect fastai loss selection and configure target values so a binary model uses CrossEntropyLoss().
Natural Language Processing in Action, Second Edition
Log loss is a probabilistic prediction metric based on the negative logarithm of the probability assigned to the observed target.
- How it is used
- Trains a binary classifier by measuring probability predictions against observed labels.
- How deep it goes
- Explains the logarithmic penalty for confident errors and shows frequency weighting and a PyTorch implementation.
- Examples
- Uses name-based sex classification and weights examples by their dataset frequencies.
- What you can do after
- Apply PyTorch BCELoss with frequency weights to train a sigmoid-output binary classifier.
Read how Natural Language Processing in Action, Second Edition explains cross entropy loss →
Deep Learning with R, Third Edition
Categorical cross-entropy is a multiclass classification loss that measures the negative log-likelihood of the target class under predicted class probabilities and expects categorically encoded labels.
- How it is used
- Guides loss selection by matching categorical cross-entropy to many-class classification and binary cross-entropy to two-class tasks.
- How deep it goes
- Defines cross-entropy as a distance between distributions and gives task-selection guidance, but no categorical-specific formula or implementation.
- Examples
- Contrasts many-class and two-class classification as loss-selection cases, without showing a categorical-cross-entropy code example.
- What you can do after
- Apply it as the loss for a many-class classification task.
Read how Deep Learning with R, Third Edition explains cross entropy loss →
Build a Reasoning Model (From Scratch)
A loss function that measures the average negative log probability assigned to the correct target tokens.
- How it is used
- Uses answer-only token loss as the training signal and validation metric during reasoning-model distillation.
- How deep it goes
- Explains token alignment, answer masking, length normalization, direct computation, and multi-example evaluation without covering optimizer internals.
- Examples
- Demonstrates the Germany–Berlin sequence, answer-token slicing, single-example loss, and averaged train-versus-validation results.
- What you can do after
- Build it from scratch with shifted token tensors, answer-boundary slicing, PyTorch cross_entropy, and aggregate train or validation evaluation.
Read how Build a Reasoning Model (From Scratch) explains cross entropy loss →
Experimentation for Engineers
A loss metric that measures the discrepancy between a probability distribution predicted by a model and the observed outcome distribution.
- How it is used
- Uses out-of-sample cross-entropy to assess prediction quality during model improvement.
- How deep it goes
- Defines the metric through a production example but stops short of giving its formula or implementation.
- Examples
- Uses video swipe probabilities, feature-based ranking, successive model versions, and DAU outcomes.
- What you can do after
- Evaluate whether prediction improvements translate into the intended business metric rather than assuming correlation.
Read how Experimentation for Engineers explains cross entropy loss →
Grokking AI Algorithms, Second Edition
Cross-entropy loss is a prediction-error measure computed as the negative logarithm of the probability assigned to the correct outcome.
- How it is used
- Measures how wrong a language model’s next-token predictions are during training.
- How deep it goes
- Explains the negative-log behavior and confidence examples, then stops short of detailing full gradient derivations.
- Examples
- Uses probabilities of 0.99, 0.50, and 0.01, plus a token-level loss table.
- What you can do after
- Evaluate token predictions by translating correct-token probabilities into interpretable error values.
Read how Grokking AI Algorithms, Second Edition explains cross entropy loss →
CUDA for LLMs
Cross-entropy loss measures the discrepancy between a target probability distribution and a predicted probability distribution.
- How it is used
- Uses cross-entropy with output softmax for classification tasks.
- How deep it goes
- Derives the combined softmax–cross-entropy gradient using probabilities, one-hot labels, and batch size.
- Examples
- Uses a one-hot target with value 1 for the correct class and 0 for all others.
- What you can do after
- Build the combined output gradient from predicted probabilities, one-hot labels, and batch size.
Also covered in
- Ensemble Methods for Machine Learning
- LLMs in Production
- Deep Learning with JAX
- Data Without Labels
- Machines that ThinkIn early access (MEAP)
- Building Reliable AI Systems
- Time Series Forecasting Using Foundation Models
- Build a DeepSeek Model (From Scratch)In early access (MEAP)
What to know first
- objective functionAn objective function is a mathematical function that assigns values to candidate solutions or parameter settings, which an optimization procedure seeks to minimize or maximize.
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- classifierA classifier is a predictive model that assigns an input to one or more discrete classes or categories.
- probability distributionA probability distribution is a mathematical specification assigning probabilities or probability densities to the possible values or outcomes of a random variable.
Related concepts
- objective functionkind of · An objective function is a mathematical function that assigns values to candidate solutions or parameter settings, which an optimization procedure seeks to minimize or maximize.
- probability distributionuses · A probability distribution is a mathematical specification assigning probabilities or probability densities to the possible values or outcomes of a random variable.
- negative log-likelihoodequivalent to · Negative log-likelihood is the negative logarithm of the likelihood a probabilistic model assigns to observed data, commonly used as an objective or loss to minimize.
- classifieraddresses · A classifier is a predictive model that assigns an input to one or more discrete classes or categories.
- focal lossincludes · Focal loss is a classification loss that downweights well-classified examples and emphasizes difficult or misclassified examples, especially under class imbalance.
- likelihoodequivalent to · A likelihood is the probability of observed data under a model, viewed as a function of the model’s parameters or candidate hypotheses.
Questions about cross entropy loss
What is cross entropy loss?
Cross-entropy loss measures the expected negative log probability that a predicted distribution assigns to outcomes drawn from a target distribution.
Which Manning books explain cross entropy loss?
Math and Architectures of Deep Learning, Grokking Machine Learning, Inside Deep Learning: Math, Algorithms, Models, Grokking Machine Learning, Second Edition, Timeless Algorithms, The Foundational Ideas, Build AI Drug Discovery Pipelines, Learn Generative AI with PyTorch, Graph Neural Networks in Action and 18 more.
What should I know before learning cross entropy loss?
Manning books that teach cross entropy loss usually introduce objective function, language model, classifier and probability distribution first.
What kinds of cross entropy loss are there?
Kinds of cross entropy loss include focal loss.
What is cross entropy loss a kind of?
Cross entropy loss is a kind of objective function. An objective function is a mathematical function that assigns values to candidate solutions or parameter settings, which an optimization procedure seeks to minimize or maximize.
What is cross entropy loss also called?
Cross entropy loss is also called CrossEntropyLoss or log loss.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

























