liveBook · Manning

Cross entropy loss

As explained in Building Specialized AI Systems

Cross-entropy loss measures the discrepancy between target outcomes and a predicted probability distribution, penalizing low probability assigned to the correct outcome.

Cross-entropy loss measures how wrong a model’s predicted probability distribution is for a target outcome. It returns one error value: smaller is better, while assigning low probability to the correct outcome produces a larger penalty. section 3.1.2

Sentiment classification

For a sentiment model, the correct label is “positive,” but the model assigns it 30% softmax probability. Cross-entropy penalizes this uncertainty and signals that the model should become more confident. section 3.1.2

During fine-tuning, cross-entropy supports the training loop by:

  • Converting model outputs and targets into a single loss value. section 3.1.2
  • Providing the error signal that gradient descent uses to adjust parameters toward lower-loss configurations. section 3.1.2
Listing 3.1 Basic steps of the training process
for inputs, targets in dataloader: #A
    outputs = model(inputs) #B
    loss = loss_fn(outputs, targets) #C
 
    optimizer.zero_grad() #D
    loss.backward() #E
    optimizer.step() #F
    if step % 10 == 0:                                   #G
        print(f"Step {step}, Loss: {loss.item():.4f}")

Shows where the loss function fits between model inference, target comparison, backpropagation, and optimizer updates. section 3.1.2

How Building Specialized AI Systems covers it

How deep it goes
Explains scalar error, probability-based penalties, loss landscapes, gradients, and the training-loop implementation.
Examples
Uses sentiment classification with a correct positive label receiving 30% probability and a basic training-loop listing.
What you can do after
Apply it via a training loop to measure prediction error and drive parameter updates during fine-tuning.
How it connects
Links loss functions with fine-tuning, softmax probabilities, gradient descent, backpropagation, and model parameters.

Also explained in

Compare how 26 books explain cross entropy loss →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Building Specialized AI Systems. The section links point to the parts of the book it draws on.