liveBook · Manning

Chain rule

As explained in Grokking AI Algorithms, Second Edition

The chain rule is a calculus rule for differentiating a composition of functions by multiplying the derivatives of its successive functions.

The chain rule finds the derivative of a composite function: a function formed by putting one function inside another. Think of a domino effect: a change in the input affects an intermediate value, which then affects the final result. Multiply the derivatives of those successive steps to find the overall effect. section 9.8 section 9.26

For example, if h(x) = f(g(x)), then h′(x) = f′(g(x)) × g′(x). The first factor describes how f changes at the output of g; the second describes how g changes with x. section 9.8

Tracing a weight’s effect on error

In a neural network, a weight can affect a hidden layer, which affects the predicted output, which affects the cost. Backpropagation follows this chain backward. For a single path, multiply the cost’s sensitivity to the prediction, the prediction’s sensitivity to the hidden value, and the hidden value’s sensitivity to the weight. The result indicates how the cost changes with that weight. If it is positive, decreasing the weight reduces cost locally; if negative, increasing it reduces cost locally. section 9.26

The backward pass starts with the output error and the slope of the activation function. It then calculates gradients: collections of partial derivatives for the weights in a layer. Gradient descent uses these slopes to guide weight updates. section 9.26 section 9.8

Figure 9.27 Weight-update calculation with the Chain Rule
09_27

Follow a worked chain-rule calculation for network weight updates. section 9.8

How Grokking AI Algorithms, Second Edition covers it

How deep it goes
Explains composite-function derivatives and traces a backward pass from output error to weight gradients.
Examples
Works through neural-network weight updates and provides a Python training loop with sigmoid activation.
What it assumes
Introduces derivatives as slopes and composite functions as functions nested inside other functions.
What you can do after
Build a simple neural-network training loop that uses backpropagation to update weights.
How it connects
Connects the chain rule to derivatives, backpropagation, activation functions, gradients, and gradient descent.

Also explained in

Compare how 4 books explain chain rule →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking AI Algorithms, Second Edition. The section links point to the parts of the book it draws on.