Cross entropy loss

As explained in Build a Large Language Model (From Scratch)

A measure of how far a language model's predicted token distribution is from the target token distribution, used to score predictions during training and evaluation.

Cross-entropy loss scores how well predicted token probabilities match target token IDs. Lower loss means the model assigns higher probability to the required next tokens. section 5.1.2

For causal language modeling, inputs and targets have matching shapes, with targets shifted one position forward. A GPT model produces logits shaped [batch, tokens, vocabulary], softmax converts them to probabilities, and the target probability is selected at each position. section 5.1.2

The manual calculation follows a short sequence:

• Select the probability assigned to each target token from the model's output distribution. section 5.1.2
• Take logarithms of those probabilities, average the log probabilities, and multiply by negative one. section 5.1.2
• Flatten logits from [batch, tokens, vocabulary] to [batch × tokens, vocabulary] and targets to [batch × tokens] before calling PyTorch cross_entropy. section 5.1.2 section 5.1.3

Worked language-model example

With two sequences of three input tokens and a vocabulary of 50,257 tokens, the probability tensor has shape [2, 3, 50257], while targets have shape [2, 3]. Flattening produces [6, 50257] logits and six target IDs. PyTorch returns a loss of 10.7940, matching the manual negative average log probability. section 5.1.2

Training and validation evaluation applies the same batch loss across data loaders, averages the batch values, and uses no-grad evaluation. A small tokenized dataset can produce separate training and validation losses before pretraining. section 5.1.3

For classification fine-tuning, cross-entropy remains the optimization target because accuracy is not differentiable. The loss uses only the final-token logits, shaped [batch, classes], against one class label per input. section 6.6

Figure 5.4 For each of the three input tokens, shown on the left, we compute a vector containing probability scores corresponding to each token in the vocabulary. The index position of the highest probability score in each vector represents the most likely next token ID. These token IDs associated with the highest probability scores are selected and mapped back into a text that represents the text generated by the model.

Trace the flow from each input token through vocabulary probabilities to the selected next-token prediction. section 5.1.2

How Build a Large Language Model (From Scratch) covers it

How deep it goes

Works through target indexing, logarithms, averaging, tensor flattening, and loader-level aggregation without deriving the loss formally.

Examples

Demonstrates shifted token batches, a 50,257-token vocabulary, manual probability scoring, and spam classification with final-token logits.

What it assumes

Assumes token IDs, shifted targets, model logits, tensor shapes, data loaders, and basic PyTorch operations.

What you can do after

Apply it via PyTorch to calculate batch, training, validation, test, language-model, and classification losses.

How it connects

Connects logits, softmax, tokenization, next-token prediction, data loaders, classification accuracy, and model fine-tuning.

liveBookmanning.com

Read Build a Large Language Model (From Scratch) on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Math and Architectures of Deep LearningA classification loss that measures the mismatch between a predicted probability distribution over classes and the target distribution.
• 

  liveBookmanning.com
  Grokking Machine LearningA multiclass classification loss that compares one-hot class labels with predicted class probabilities using cross-entropy.
• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA differentiable classification loss that measures the mismatch between predicted and target probability distributions.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorchA loss measure that compares predicted token distributions with the target next tokens for multiclass sequence prediction.
• 

  liveBookmanning.com
  Graph Neural Networks in ActionCrossEntropyLoss is a PyTorch loss module that measures classification error from logits and target class labels.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA loss value based on cross-entropy, used here as the quantity minimized during training.
• 

  liveBookmanning.com
  Knowledge Graphs and LLMs in ActionCross-entropy loss is a classification loss that penalizes incorrect probability assignments by comparing predicted probabilities with target labels.
• 

  liveBookmanning.com
  Reinforcement Learning from Human FeedbackCross-entropy is the expected negative log probability assigned by one distribution to outcomes drawn from another distribution.
• 

  liveBookmanning.com
  Machine Learning System DesignA classification loss that penalizes incorrect probabilistic predictions using the negative logarithm of the probability assigned to the true class.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA classification loss that measures the mismatch between predicted probability distributions and true token labels.
• 

  liveBookmanning.com
  How Large Language Models WorkA cross-entropy loss function measures the discrepancy between a target probability distribution and a model's predicted probability distribution.
• 

  liveBookmanning.com
  Build a Text-to-Image Generator (from Scratch)Categorical cross-entropy loss measures the discrepancy between predicted class probabilities and the ground-truth class in multicategory classification.
• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AICrossEntropyLoss is a PyTorch loss-function module that combines class-logit normalization with negative log-likelihood for classification.
• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionLog loss is a probabilistic prediction metric based on the negative logarithm of the probability assigned to the observed target.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionCategorical cross-entropy is a multiclass classification loss that measures the negative log-likelihood of the target class under predicted class probabilities and expects categorically encoded labels.
• 

  liveBookmanning.com
  Build a Reasoning Model (From Scratch)A loss function that measures the average negative log probability assigned to the correct target tokens.
• 

  liveBookmanning.com
  Experimentation for EngineersA loss metric that measures the discrepancy between a probability distribution predicted by a model and the observed outcome distribution.
• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionCross-entropy loss is a prediction-error measure computed as the negative logarithm of the probability assigned to the correct outcome.

Compare how 19 books explain cross entropy loss →

Related concepts

• classifieraddresses · A classifier is a predictive computational model that assigns inputs to one or more discrete classes or categories.
• negative log-likelihoodequivalent to · Negative log-likelihood is the negative logarithm of the likelihood a probabilistic model assigns to observed data, commonly used as an objective or loss to minimize.
• focal lossincludes · Focal loss is a classification loss that downweights well-classified examples and emphasizes difficult or misclassified examples, especially under class imbalance.
• uncertaintymeasures · Uncertainty is incomplete knowledge or confidence about a value, state, interpretation, behavior, or outcome.
• likelihoodequivalent to · A likelihood is the probability of observed data under a specified probabilistic model, regarded as a function of candidate hypotheses or parameter values.

This explainer was generated by Manning's knowledge graph from the text of Build a Large Language Model (From Scratch). The section links point to the parts of the book it draws on.