Chain rule

As explained in Deep Learning with PyTorch, Second Edition

A differentiation rule for computing the derivative of a composite function by multiplying derivatives along the composition.

The chain rule computes how an outer result changes when an inner quantity changes. For a loss built from a model, multiply the loss derivative with respect to the model output by the model derivative with respect to each parameter. section 5.4.2

In learning, this turns a change in model parameters into a change in loss. The resulting derivatives form a gradient, which indicates how to adjust weights and bias to reduce the error. section 5.2 section 5.4.2

For a linear model followed by a mean-squared-error loss, the calculation proceeds as follows:

• Differentiate the loss with respect to each prediction; averaging contributes a division by the number of predictions. section 5.4.2

Manual gradient for a model and loss

The implementation multiplies derivatives directly: dloss_dw = dloss_dtp * dmodel_dw(t_u, w, b) and dloss_db = dloss_dtp * dmodel_db(t_u, w, b). This is a simple backward pass through the loss and model. section 5.5

Figure 5.6 Comparing numerical vs. analytical gradient calculation. The numerical approach (arrow through points) estimates the gradient using finite differences between discrete points, while the analytical approach (arrow along the curves) calculates the exact gradient at a specific point.

Compare numerical gradient estimates between discrete points with analytical derivatives evaluated at a specific point. section 5.4.2

How Deep Learning with PyTorch, Second Edition covers it

How deep it goes

Works through analytical derivatives for a linear model and squared-error loss, then extends the idea to differentiable deep compositions.

Examples

Uses a linear model, mean-squared-error loss, manual derivative functions, and a combined gradient function.

What it assumes

Assumes basic model, prediction, loss, and parameter ideas while developing differentiation through a simple learning problem.

What you can do after

Apply it to compute parameter gradients manually and understand how automatic differentiation performs the same backward propagation.

How it connects

Links derivatives, gradients, loss functions, model parameters, backpropagation, and automatic differentiation.

liveBookmanning.com

Read Deep Learning with PyTorch, Second Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionThe chain rule is a calculus rule for differentiating a composition of functions by multiplying the derivatives of its successive functions.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionThe chain rule is a calculus rule for differentiating a composition of functions by multiplying derivatives of the component functions.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionThe differentiation rule for composing functions, used to compute gradients through layered neural-network operations.

Compare how 4 books explain chain rule →

Related concepts

• backpropagationused by · Backpropagation is a procedure for computing derivatives of an output or loss with respect to inputs and parameters by applying the chain rule backward through a computational graph.
• derivativeproduces · A derivative is a mathematical quantity that measures the instantaneous rate of change of a function with respect to an input variable.
• automatic differentiationused by · Automatic differentiation is a computational technique that evaluates derivatives of a program by systematically applying the chain rule to its elementary operations.
• forward-mode autodiffused by · An automatic differentiation mode that propagates derivatives forward from inputs to outputs and is efficient when there are few inputs relative to outputs.
• gradientuses · A gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.
• Jacobianuses · The matrix of all first-order partial derivatives of a vector-valued function with respect to its inputs.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with PyTorch, Second Edition. The section links point to the parts of the book it draws on.