The chain rule finds how a change travels through a sequence of functions. If one value changes a second value, which changes a third, multiply the derivatives for each step to find the overall effect. It is a way to follow a mathematical domino effect. section 9.26 section 9.8
In a neural network, a weight does not change the cost directly. It changes a layer’s value, which changes the prediction, which changes the cost. Forward propagation computes the prediction; backpropagation follows those dependencies backward to find how much each weight contributed to the error. The resulting partial derivative, ∂J/∂W, describes how cost J changes with one weight W. section 9.26
Follow one weight’s effect
Consider a connection from a hidden unit to an output unit. Its weight affects the output calculation, the activation function turns that calculation into a prediction, and the prediction affects cost. Starting at the cost, work backward: find the prediction’s error, account for the activation function’s slope, then account for the hidden unit’s value. Multiplying these effects gives that weight’s cost gradient. For a layer of weights, the same calculation can be organized as a matrix operation: multiply the transpose of the hidden-layer values by the output error. section 9.26
Use the result during training:
- Run forward propagation to obtain a prediction and compare it with the expected output. section 9.9
- Work backward through the layers, multiplying each step’s derivative to calculate weight gradients. section 9.26
- Adjust the weights to reduce cost. A positive ∂J/∂W calls for decreasing that weight; a negative value calls for increasing it. section 9.26 section 9.8
Figure 9.26 Formula for calculating weight updates with the Chain Rule
The weight-update diagram shows where the chain rule enters the neural-network training calculation. section 9.8
In Python, this backward calculation needs derivatives of the activation functions. For sigmoid, a learning implementation can compute the derivative as sigmoid(x) × (1 − sigmoid(x)), then alternate forward propagation and backpropagation across training epochs. The chain rule supplies the gradients; gradient descent uses them to choose weight adjustments. Other optimization approaches can also be used. section 9.9 section 9.8
