The chain rule differentiates a composition of functions by multiplying the derivatives of its component operations. In neural networks, it turns the derivatives of simple operations such as addition, ReLU, tensor products, softmax, and loss functions into gradients for complex models. section 2.4.4
From a two-layer network to backpropagation
A two-layer network applies matrix multiplication, addition, ReLU, another matrix multiplication, addition, softmax, and a loss function. The chain rule passes the loss gradient backward through these operations, multiplying local derivatives along the path to each weight. This procedure is called backpropagation and supplies the gradients used to update the model. section 2.4.4 section 2.5
fg <- function(x) {
x1 <- g(x)
y <- f(x1)
y
}
Trace a composition explicitly by naming the intermediate value before applying the next function. section 2.4.4
fghj <- function(x) {
x1 <- j(x)
x2 <- h(x1)
x3 <- g(x2)
y <- f(x3)
y
}
grad(y, x) == (grad(y, x3) * grad(x3, x2) * grad(x2, x1) * grad(x1, x))
Extend the two-function pattern to a longer chain and read the gradient as a product of derivatives through successive intermediate values. section 2.4.4
Computation graphs make the chain visible: a forward pass computes the output, while a backward pass follows paths from the loss to the parameters. Mini-batch gradient descent then uses those gradients to move weights toward lower loss. section 2.4.4 section 2.6
