Cross entropy loss
Cross-entropy loss measures the expected negative log probability that a predicted distribution assigns to outcomes drawn from a target distribution.
Math and Architectures of Deep Learning
A classification loss that measures the mismatch between a predicted probability distribution over classes and the target distribution.
- How it is used
- Measures classifier-output mismatch against one-hot ground truth during training.
- How deep it goes
- Derives the multiclass formula, reduces it to the correct-class log probability, and explains softmax integration.
- Examples
- Uses cat, dog, airplane, and automobile predictions, plus height-and-weight and region-of-interest classifiers.
- What you can do after
- Apply it via PyTorch cross_entropy or CrossEntropyLoss to raw classifier scores and ground-truth labels.
Read how Math and Architectures of Deep Learning explains cross entropy loss →
Grokking Machine Learning
A multiclass classification loss that compares one-hot class labels with predicted class probabilities using cross-entropy.
- How it is used
- Measures how much probability a classifier assigns to the correct label, serving as the loss for multiclass neural-network training.
- How deep it goes
- Works through binary log loss, then identifies and configures its multiclass version without deriving multiclass training updates.
- Examples
- Uses happy-or-sad sentence predictions, a Keras classifier, and one-hot labels for handwritten digits.
- What you can do after
- Apply it via Keras by compiling a softmax classifier with categorical_crossentropy and monitoring loss alongside accuracy.
Read how Grokking Machine Learning explains cross entropy loss →
Inside Deep Learning: Math, Algorithms, Models
A differentiable classification loss that measures the mismatch between predicted and target probability distributions.
- How it is used
- Trains classification models by turning scores into probabilities and penalizing mismatch with target labels.
- How deep it goes
- Explains score-to-probability conversion, target distributions, confidence intuition, and PyTorch binary-loss implementation.
- Examples
- Uses one-hot class distributions, GAN real-versus-fake labels, and Seq2Seq output sequences as teaching examples.
- What you can do after
- Apply it via PyTorch's BCEWithLogitsLoss when training a binary GAN discriminator and generator.
Read how Inside Deep Learning: Math, Algorithms, Models explains cross entropy loss →
Learn Generative AI with PyTorch
A loss measure that compares predicted token distributions with the target next tokens for multiclass sequence prediction.
- How it is used
- Trains multicategory classifiers by penalizing disagreement between predicted distributions and target labels or next tokens.
- How deep it goes
- Explains the loss concept, logits interface, internal LogSoftmax and NLLLoss combination, padding exclusion, and training-loop placement.
- Examples
- Uses ten-class clothing classification and next-token music generation as worked applications.
- What you can do after
- Apply it via PyTorch to train classifiers and Transformers, including padded token sequences with an ignored target index.
Read how Learn Generative AI with PyTorch explains cross entropy loss →
Graph Neural Networks in Action
CrossEntropyLoss is a PyTorch loss module that measures classification error from logits and target class labels.
- How it is used
- Uses log loss to compare multiclass GCN and GraphSAGE predictions and validation performance.
- How deep it goes
- Works through loss curves and metric comparisons, but does not specify CrossEntropyLoss inputs, implementation, or a mathematical derivation.
- Examples
- Compares GCN and GraphSAGE validation scores, then contrasts mean, sum, max, and list aggregation results.
- What you can do after
- Evaluate/understand model trade-offs by reading validation-loss curves and comparing log loss with F1 score.
Read how Graph Neural Networks in Action explains cross entropy loss →
Build a Large Language Model (From Scratch)
A measure of how far a language model's predicted token distribution is from the target token distribution, used to score predictions during training and evaluation.
- How it is used
- Uses prediction error to evaluate and train language models, then reuses the objective for classification fine-tuning.
- How deep it goes
- Works through target indexing, logarithms, averaging, tensor flattening, and loader-level aggregation without deriving the loss formally.
- Examples
- Demonstrates shifted token batches, a 50,257-token vocabulary, manual probability scoring, and spam classification with final-token logits.
- What you can do after
- Apply it via PyTorch to calculate batch, training, validation, test, language-model, and classification losses.
Read how Build a Large Language Model (From Scratch) explains cross entropy loss →
Deep Learning with PyTorch, Second Edition
A loss value based on cross-entropy, used here as the quantity minimized during training.
- How it is used
- Measures classification error so training can adjust class predictions.
- How deep it goes
- Explains its equivalence to LogSoftmax plus NLLLoss and compares gradient behavior with MSE, without deriving the loss mathematically.
- Examples
- Uses bird-versus-airplane predictions, a plotted loss comparison, and a two-class nodule dataset.
- What you can do after
- Apply it via PyTorch by supplying logits directly, or use LogSoftmax with NLLLoss instead.
Read how Deep Learning with PyTorch, Second Edition explains cross entropy loss →
Knowledge Graphs and LLMs in Action
Cross-entropy loss is a classification loss that penalizes incorrect probability assignments by comparing predicted probabilities with target labels.
- How it is used
- Trains graph models for link prediction and node classification by comparing predicted probabilities with relationships or labels.
- How deep it goes
- Explains sigmoid, negative sampling, probability penalties, and log_softmax integration, but does not derive the full optimization mathematics.
- Examples
- Uses an aspirin-treats-headache knowledge-graph fact and licit-versus-illicit transaction classification.
- What you can do after
- Apply PyTorch CrossEntropyLoss with log_softmax outputs to train a GNN classifier and evaluate label mismatches.
Read how Knowledge Graphs and LLMs in Action explains cross entropy loss →
Reinforcement Learning from Human Feedback
Cross-entropy is the expected negative log probability assigned by one distribution to outcomes drawn from another distribution.
- How it is used
- Trains language models, distilled students, and process reward models by matching predicted probabilities to labeled or teacher-provided targets.
- How deep it goes
- Defines distribution matching, derives its KL relationship, and reaches masked PyTorch computation without developing optimization internals.
- Examples
- Uses teacher-token distributions, step-boundary labels, separator tokens, masked labels, and a PyTorch functional cross-entropy call.
- What you can do after
- Apply it via masked PyTorch computation or interpret it as forward-KL matching in supervised and distillation objectives.
Read how Reinforcement Learning from Human Feedback explains cross entropy loss →
Machine Learning System Design
A classification loss that penalizes incorrect probabilistic predictions using the negative logarithm of the probability assigned to the true class.
- How it is used
- Uses binary cross-entropy to train search systems to predict clicks and purchases.
- How deep it goes
- Contrasts binary cross-entropy with focal loss for imbalanced data, without detailing the loss computation.
- Examples
- Uses search clicks and purchases as prediction targets, and spam versus nonspam as a classification example.
- What you can do after
- Apply binary cross-entropy to train click and purchase predictors, then compare results with online metrics such as CTR and conversion rate.
Read how Machine Learning System Design explains cross entropy loss →
Deep Learning with Python, Third Edition
A classification loss that measures the mismatch between predicted probability distributions and true token labels.
- How it is used
- Guides loss selection for classification tasks with probability outputs.
- How deep it goes
- Defines cross-entropy as a distance between distributions and gives task-based loss choices without deriving the calculation.
- Examples
- Uses binary classifiers for sentiment and dogs-versus-cats classification, alongside a categorical-classification rule.
- What you can do after
- Apply it via Keras by selecting binary or categorical cross-entropy to match the classification task.
Read how Deep Learning with Python, Third Edition explains cross entropy loss →
How Large Language Models Work
A cross-entropy loss function measures the discrepancy between a target probability distribution and a model's predicted probability distribution.
- How it is used
- Replaces accuracy during training with a better-behaved proxy objective.
- How deep it goes
- Explains its proxy role but defers mathematical details and implementation.
- Examples
- Uses cancer-case counts and smooth-versus-nonsmooth function figures to illustrate the training problem.
- What you can do after
- Evaluate why training can optimize a proxy rather than the desired behavior.
Read how How Large Language Models Work explains cross entropy loss →
Build a Text-to-Image Generator (from Scratch)
Categorical cross-entropy loss measures the discrepancy between predicted class probabilities and the ground-truth class in multicategory classification.
- How it is used
- Trains multicategory classifiers, caption decoders, translation models, and text-to-image systems by minimizing prediction error.
- How deep it goes
- Explains target alignment and parameter minimization, then implements the loss with PyTorch without deriving its mathematical formula.
- Examples
- Uses image classes, the caption “a dog runs,” German–English translation batches, and discrete image-token sequences.
- What you can do after
- Apply it via PyTorch to train classifiers and autoregressive models against categorical or token-level targets.
Read how Build a Text-to-Image Generator (from Scratch) explains cross entropy loss →
Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI
CrossEntropyLoss is a PyTorch loss-function module that combines class-logit normalization with negative log-likelihood for classification.
- How it is used
- Selects the training objective for categorical prediction and helps verify that a tabular model is solving classification rather than regression.
- How deep it goes
- Defines the classification role and demonstrates loss-function diagnosis, but does not derive the calculation mathematically.
- Examples
- Uses binary Airbnb price classification, numeric-versus-string targets, MSELoss(), and CrossEntropyLoss() inspection.
- What you can do after
- Diagnose incorrect fastai loss selection and configure target values so a binary model uses CrossEntropyLoss().
Natural Language Processing in Action, Second Edition
Log loss is a probabilistic prediction metric based on the negative logarithm of the probability assigned to the observed target.
- How it is used
- Trains a binary classifier by measuring probability predictions against observed labels.
- How deep it goes
- Explains the logarithmic penalty for confident errors and shows frequency weighting and a PyTorch implementation.
- Examples
- Uses name-based sex classification and weights examples by their dataset frequencies.
- What you can do after
- Apply PyTorch BCELoss with frequency weights to train a sigmoid-output binary classifier.
Read how Natural Language Processing in Action, Second Edition explains cross entropy loss →
Deep Learning with R, Third Edition
Categorical cross-entropy is a multiclass classification loss that measures the negative log-likelihood of the target class under predicted class probabilities and expects categorically encoded labels.
- How it is used
- Guides loss selection by matching categorical cross-entropy to many-class classification and binary cross-entropy to two-class tasks.
- How deep it goes
- Defines cross-entropy as a distance between distributions and gives task-selection guidance, but no categorical-specific formula or implementation.
- Examples
- Contrasts many-class and two-class classification as loss-selection cases, without showing a categorical-cross-entropy code example.
- What you can do after
- Apply it as the loss for a many-class classification task.
Read how Deep Learning with R, Third Edition explains cross entropy loss →
Build a Reasoning Model (From Scratch)
A loss function that measures the average negative log probability assigned to the correct target tokens.
- How it is used
- Uses answer-only token loss as the training signal and validation metric during reasoning-model distillation.
- How deep it goes
- Explains token alignment, answer masking, length normalization, direct computation, and multi-example evaluation without covering optimizer internals.
- Examples
- Demonstrates the Germany–Berlin sequence, answer-token slicing, single-example loss, and averaged train-versus-validation results.
- What you can do after
- Build it from scratch with shifted token tensors, answer-boundary slicing, PyTorch cross_entropy, and aggregate train or validation evaluation.
Read how Build a Reasoning Model (From Scratch) explains cross entropy loss →
Experimentation for Engineers
A loss metric that measures the discrepancy between a probability distribution predicted by a model and the observed outcome distribution.
- How it is used
- Uses out-of-sample cross-entropy to assess prediction quality during model improvement.
- How deep it goes
- Defines the metric through a production example but stops short of giving its formula or implementation.
- Examples
- Uses video swipe probabilities, feature-based ranking, successive model versions, and DAU outcomes.
- What you can do after
- Evaluate whether prediction improvements translate into the intended business metric rather than assuming correlation.
Read how Experimentation for Engineers explains cross entropy loss →
Grokking AI Algorithms, Second Edition
Cross-entropy loss is a prediction-error measure computed as the negative logarithm of the probability assigned to the correct outcome.
- How it is used
- Measures how wrong a language model’s next-token predictions are during training.
- How deep it goes
- Explains the negative-log behavior and confidence examples, then stops short of detailing full gradient derivations.
- Examples
- Uses probabilities of 0.99, 0.50, and 0.01, plus a token-level loss table.
- What you can do after
- Evaluate token predictions by translating correct-token probabilities into interpretable error values.
Read how Grokking AI Algorithms, Second Edition explains cross entropy loss →
Related concepts
- classifieraddresses · A classifier is a predictive computational model that assigns inputs to one or more discrete classes or categories.
- negative log-likelihoodequivalent to · Negative log-likelihood is the negative logarithm of the likelihood a probabilistic model assigns to observed data, commonly used as an objective or loss to minimize.
- focal lossincludes · Focal loss is a classification loss that downweights well-classified examples and emphasizes difficult or misclassified examples, especially under class imbalance.
- uncertaintymeasures · Uncertainty is incomplete knowledge or confidence about a value, state, interpretation, behavior, or outcome.
- likelihoodequivalent to · A likelihood is the probability of observed data under a specified probabilistic model, regarded as a function of candidate hypotheses or parameter values.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.