The chain rule computes the derivative of a composed function from the derivatives of its smaller steps. This matters because neural networks chain tensor operations, allowing gradients to flow backward through complex models. section 2.4.4
From two functions to a gradient chain
Let x1 = g(x) and y = f(x1). Then grad(y, x) = grad(y, x1) * grad(x1, x). Add more operations and the factors extend into a chain: grad(y, x) = grad(y, x3) * grad(x3, x2) * grad(x2, x1) * grad(x1, x). section 2.4.4
loss_value = loss( y_true, softmax(matmul(relu(matmul(inputs, W1) + b1), W2) + b2), )
Trace a two-layer network as one composition: matrix multiplication, addition, ReLU, matrix multiplication, addition, softmax, and loss. section 2.4.4
Use the rule as a practical checklist:
- Break a complex expression into atomic operations with known derivatives. section 2.4.4
- Multiply local derivatives along the computation path to obtain the overall gradient. section 2.4.4
