Backpropagation trains a neural network by measuring its error, calculating how every weight and bias contributed to that error, and adjusting them to improve future predictions. It combines the intuition of correcting connected classifiers with gradient descent: the loss gradient points toward increasing error, so training takes a small step in the opposite direction. section 10.2 section 10.6
A training iteration follows this compact procedure:
- Initialize the network with random weights and biases. section 10.2
- Calculate predictions, the loss function, and its derivatives with respect to every weight and bias. section 10.2
- Update the parameters by taking a small step opposite the gradient. section 10.2
- Repeat the process across epochs and batches while monitoring loss and accuracy. section 10.3
A small classifier in Keras
A two-feature dataset is classified with two hidden layers, ReLU activations, dropout, and a softmax output. Calling model.fit(..., epochs=100, batch_size=10) runs repeated backpropagation; training loss falls from 0.5473 to 0.2110 while accuracy reaches 0.9000. Testing accuracy should still be measured separately. section 10.3
Figure 10.16 A mental picture of backpropagation. At each step of the training process, the weights of the edges are updated. If a classifier is good, its weight gets increased by a small amount, and if it is bad, its weight gets decreased.
Visualizes backpropagation as repeated edge-weight updates: useful weights increase slightly and unhelpful weights decrease slightly. section 10.2
