Backpropagation helps a neural network learn from a wrong prediction. Imagine a relay team that finishes late: to find what went wrong, a coach works backward from the finish, considering how each runner contributed to the delay. Backpropagation similarly starts with a loss score—a measure of prediction error—and works backward to find how each weight contributed. section 9.5
A training cycle follows this order:
- Run a forward pass to make a prediction, then compare it with the expected result to calculate loss. section 11.32
- Work backward through the layers. Use the chain rule to calculate each weight’s gradient: how changing that weight would affect the loss. section 9.26
- Use the gradients and a learning rate to adjust the weights. Repeat with more data over successive training steps. section 11.32 section 12.22
Why work backward? A weight deep in a network does not affect the loss directly: it changes a layer’s output, which changes later outputs, which changes the prediction. The chain rule links those effects. A positive gradient means increasing that weight increases error, so the update should decrease it; a negative gradient points the other way. Backpropagation calculates these gradients, while the weight update uses them to improve the next attempt. section 9.26 section 11.32
One weight update
Suppose a weight is 0.5, its calculated gradient is −0.5, and the learning rate is 0.01. The update increases the weight by 0.005, to 0.505. This is one small adjustment; training repeats the process across the network and over many steps. section 12.22
The same idea applies beyond a small artificial neural network. In a language model, gradients travel back through output layers, attention, and embeddings. In an image-generation model, they travel through layers of a U-Net as it learns to predict noise. A learning implementation can show the forward and backward calls in a loop, but an illustrative placeholder for gradient calculation is not a complete implementation. section 9.9 section 11.30 section 12.23
def run_neural_network(epochs, feature_data, label_data, hidden_node_count):
scaled_feature_data = scale_dataset(
feature_data, FEATURE_COUNT, FEATURE_MIN, FEATURE_MAX
) #A
nn = NeuralNetwork( #B
scaled_feature_data,
label_data,
hidden_node_count
)
for epoch in range(epochs): #C
nn.forward_propagation() #D
nn.back_propagation() #E
if epoch % 100 == 0:
loss = np.mean(np.square(nn.expected_output - nn.output)) #F
print(f"Epoch {epoch:03d}: Loss = {loss:.6f}")
return nn
The Python training loop pairs each forward pass with a backward pass and periodically prints the loss, making the repeated learning cycle concrete. section 9.9
