An objective function expresses what a learning system is trying to minimize or maximize. During model training, a loss function turns a prediction’s error into a number: the training loss. Think of it as a score for how far the model’s answer is from the answer available in its training examples. The choice of loss matters because it defines what the model is being trained to improve. section 5.22 section 9.17 section 9.8
A typical learning cycle uses loss like this:
- Make predictions from a batch of training examples, then compare them with the known answers to obtain a loss value. section 11.32
- Use backpropagation to calculate weight adjustments, update the weights, and repeat with more batches. section 11.32 section 12.22
- Watch the trend, not just one score: decreasing training loss suggests improvement on training examples, but does not establish that the model generalizes. section 12.21 section 11.34
Choose a loss that matches the prediction
For a neural network predicting a class, a simple illustration compares a prediction of 0.7 with an actual value of 1.0, giving a difference of 0.3. For a diffusion model predicting continuous noise values, mean squared error averages the squared differences between predicted and actual noise across pixels. For next-token prediction, cross-entropy assigns a small loss when the correct token received high probability and a large loss when it received low probability. section 9.8 section 12.21 section 11.30
Figure 12.46 The ideal loss-over-time for a diffusion model
The loss-over-time figure illustrates the useful pattern to look for while training: a downward trend rather than a single impressive score. section 12.21
Training loss is computed on examples the model learns from. To check whether improvement carries beyond those examples, calculate loss separately on a validation set the model never trains on. If training loss falls while validation loss rises, overfitting may be underway; early stopping uses validation loss to decide when to stop. section 11.34
