Backpropagation trains a neural network by measuring its loss, calculating how each weight and bias affects that loss, and adjusting those parameters to improve performance. section 10.11 section 10.25
A training cycle follows this pattern:
- Initialize the neural network with random weights and biases. section 10.11
- Calculate the loss and its derivatives with respect to every weight and bias. section 10.11
- Move a small step opposite the gradient so the loss decreases. section 10.11
- Repeat the process many times until the resulting parameters likely fit the data well. section 10.11
A mental picture
Focus on one node and one data point. At each training step, the network updates the weights on its edges: a useful classifier's weight increases slightly, while a poor classifier's weight decreases. section 10.11
Figure 10.16 A mental picture of backpropagation. At each step of the training process, the weights of the edges are updated. If a classifier is good, its weight gets increased by a small amount, and if it is bad, its weight gets decreased.
Use the figure to visualize edge weights changing during successive training steps. section 10.11
The loss can be complicated because predictions depend on many variables and may include logarithms. Keras, TensorFlow, and PyTorch implement backpropagation efficiently, so users mainly specify the network architecture and error functions. section 10.11 section 10.25
























