An objective function, also called a loss function, criterion, or cost function, turns the difference between a model’s output and a desired output into a number. Training adjusts the model’s parameters to make that number smaller. A low loss on training data is useful, but the goal is also to perform well on data the model has not seen. section 1.2
Imagine calibrating a thermometer with a linear model, t_p = w * t_u + b. Its objective compares each predicted Celsius temperature t_p with a measured temperature t_c. Both |t_p - t_c| and (t_p - t_c)² penalize errors in either direction; squaring gives larger errors more weight. section 5.2.4 section 5.3
Mean squared loss in PyTorch
A small loss function can compute the differences for a batch, square them, and return their mean: `def loss_fn(t_p, t_c): return ((t_p - t_c)**2).mean()`. The result is a scalar tensor. In a training loop, compute predictions, call `loss_fn`, run `loss.backward()`, and let the optimizer update the parameters. section 5.3.1 section 5.5.3
Choose the objective for the prediction task. Mean squared error suits the numerical temperatures above, but it is not well suited to categorical image labels. A binary bird-versus-airplane classifier can instead use a probability output with binary cross-entropy. A diffusion model uses mean squared error for a different target: the noise added to an image. section 7.2.3 section 10.6.1
Track training loss alongside validation loss, which is evaluated on held-out data. If training loss stops improving, the model may be too simple or the inputs may lack useful information. If training loss falls while validation loss rises, suspect overfitting. Loss is not the whole performance report: a classifier can appear accurate by predicting only the majority class, so inspect task-relevant metrics too. section 5.5.3 section 14.0
Figure 5.14 Overfitting scenarios when looking at the training (solid line) and validation (dotted line) losses. (A) Training and validation losses do not decrease; the model is not learning because there is no information in the data or the model’s capacity is insufficient. (B) Training loss decreases while validation loss increases: overfitting. (C) Training and validation losses decrease exactly in tandem. Performance may be improved further as the model is not at the limit of overfitting. (D) Training and validation losses have different absolute values but similar trends; overfitting is under control.
Compare the training and validation loss curves to distinguish failure to learn from overfitting. section 5.5.3
