Backpropagation computes how each weight contributes to a neural network’s prediction error. After a forward pass produces a prediction, a loss function compares it with the expected output. Gradients then flow backward from the output layer through the hidden layers, giving the optimizer information for adjusting weights and biases. section 8.3.4 section 8.4.1
A training loop follows these steps:
- Pass an input through the network to generate a prediction, then calculate its loss against the expected output. section 8.4.1
- Calculate loss gradients at the final layer and propagate them toward the first layer, combining derivatives with the chain rule. section 8.4.1
- Use those gradients to adjust weights and biases, then repeat. The learning rate controls the size of the corrective steps. section 8.4
The key quantity is the derivative of the loss with respect to each weight: it indicates how changing that weight affects the loss. Backpropagation calculates these derivatives layer by layer; optimization uses them to reduce loss over repeated training passes. section 8.4.1 section 8.4.2
A classification training pass
For a supervised dogs-versus-cats classifier, compare a predicted output with the known label to obtain a loss. Propagate that error backward, adjust the network parameters, and repeat before using the trained network on unseen images. A lower training loss is the aim of the updates, not a guarantee about every unseen prediction. section 8.5 section 8.4.2
Figure 8.11 Backpropagation as a process: the information flows from the final layers to the initial layers
Trace the direction of information from the final layer back through earlier layers, opposite to the forward pass. section 8.4
