liveBook · Manning

Backpropagation

As explained in Grokking Machine Learning, Second Edition

A neural-network training algorithm that computes loss gradients with respect to model parameters by propagating error backward through the network.

Backpropagation trains a neural network by measuring its loss, calculating how each weight and bias affects that loss, and adjusting those parameters to improve performance. section 10.11 section 10.25

A training cycle follows this pattern:

  • Initialize the neural network with random weights and biases. section 10.11
  • Calculate the loss and its derivatives with respect to every weight and bias. section 10.11
  • Move a small step opposite the gradient so the loss decreases. section 10.11
  • Repeat the process many times until the resulting parameters likely fit the data well. section 10.11

A mental picture

Focus on one node and one data point. At each training step, the network updates the weights on its edges: a useful classifier's weight increases slightly, while a poor classifier's weight decreases. section 10.11

Figure 10.16 A mental picture of backpropagation. At each step of the training process, the weights of the edges are updated. If a classifier is good, its weight gets increased by a small amount, and if it is bad, its weight gets decreased.

Use the figure to visualize edge weights changing during successive training steps. section 10.11

The loss can be complicated because predictions depend on many variables and may include logarithms. Keras, TensorFlow, and PyTorch implement backpropagation efficiently, so users mainly specify the network architecture and error functions. section 10.11 section 10.25

How Grokking Machine Learning, Second Edition covers it

How deep it goes
Defines the training pseudocode and intuition, while deferring deeper-network mathematics to further resources.
Examples
Uses a single node, one data point, and changing edge weights to build a mental picture of training.
What it assumes
Assumes familiarity with gradient descent while relating neural-network training to earlier regression and perceptron models.
What you can do after
Apply it via Keras, TensorFlow, or PyTorch by defining a network architecture and error function.
How it connects
Connects gradient descent, loss functions, neural networks, vanishing gradients, regularization, dropout, and optimizers.

Also explained in

Compare how 25 books explain backpropagation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning, Second Edition. The section links point to the parts of the book it draws on.