A network can nearly perfect its training answers and still fail on new ones. In a random-label experiment, a smaller AlexNet fit its training data almost perfectly, yet its test accuracy was near chance. Low training loss, then, is not proof that a model has learned a useful pattern. section 8.5.2
An objective function gives training a quantity to reduce. A loss function measures the difference between a model’s predictions and the correct answers. As the network adjusts its weights and biases, backpropagation computes how the loss changes with each weight; gradient descent uses that feedback to update the parameters. This practical training loop helped revive multilayer neural networks in the 1980s. section 2.0
AlexNet illustrates how loss becomes an engineering problem. It used mini-batch stochastic gradient descent: each batch provided a noisy estimate of which way was downhill, while momentum helped steady successive updates. The learning rate set the size of each move and was reduced when validation accuracy flattened. Minimizing the training objective required choices about batches, updates, and when to change pace. section 2.5.4
A low loss can hide a failed task
A smaller AlexNet trained on CIFAR-10 with random labels reached 99.82% training accuracy but only 9.86% test accuracy. It could fit the supplied answers without learning labels that carried over to the test set. section 8.5.2
Table 8.1 Training and test accuracy of a smaller version of AlexNet on the CIFAR-10 dataset. The original AlexNet had 60 million parameters.
|
Model
|
Number of parameters
|
Labels
|
Random crop
|
Weight decay
|
Train accuracy
|
Test accuracy
|
|---|---|---|---|---|---|---|
|
AlexNet
|
1,387,786
|
true
|
yes
|
yes
|
99.90
|
81.22
|
|
true
|
yes
|
no
|
99.82
|
79.66
|
||
|
true
|
no
|
yes
|
100.0
|
77.36
|
||
|
true
|
no
|
no
|
100.0
|
76.07
|
||
|
random
|
no
|
no
|
99.82
|
9.86
|
Compare the random-label row with the true-label rows: similarly high training accuracy accompanies sharply different test accuracy. section 8.5.2
At larger scales, training loss also serves as a forecasting signal: scaling laws relate its decline to increases in model size, data, and compute. But the relationship can change when a run encounters a new regime, and a smooth loss curve need not mean smooth gains on a benchmark. Thresholded evaluations can jump when small improvements turn borderline answers into correct ones. Loss helps guide training and compute decisions; separate evaluations show what capabilities have actually improved. section 6.0 section 6.2.1
