liveBook · Manning

Backpropagation

As explained in Machines that Think

A backward pass is the computation that propagates error information backward through a neural network to obtain parameter gradients.

Backpropagation traces a neural network’s error backward through its layers to determine how each weight contributes to the loss. These sensitivities provide the gradients used to improve the network’s predictions. section 3.2.2

A forward pass transforms inputs into an output, and the loss measures how far that output is from the desired answer. Backpropagation then asks how the loss would change if each weight changed slightly. Derivatives express those sensitivities; together, they form the gradient. An optimizer uses the gradient to adjust parameters toward lower loss. section 3.2.2 section 3.6

Modern software frameworks use automatic differentiation to calculate the derivatives needed for backward propagation. Training repeats forward and backward passes with optimization steps, while inference typically uses only a forward pass with fixed parameters. section 3.2.2 section 4.6.1

At large scale, backward passes are part of a substantial engineering workload. Gradients must be aggregated across machines, and intermediate activations must be stored or recomputed within memory limits. section 4.6.1

How Machines that Think covers it

How deep it goes
Explains weight sensitivities through derivatives and gradients, and identifies automatic differentiation as the practical computation method.
Examples
Illustrates sensitivity by asking how a slight change in a weight would affect the loss.
What you can do after
Evaluate or understand how gradients guide training and why training requires backward passes while inference typically does not.
How it connects
Connects backpropagation to the forward pass, loss, gradients, automatic differentiation, and optimization.

Also explained in

Compare how 25 books explain backpropagation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.