liveBook · Manning

Backpropagation

As explained in Deep Learning with Python, Third Edition

A PyTorch tensor method that triggers backpropagation from a loss tensor and fills gradients on participating tensors.

A loss says how far a model’s predictions are from their targets. Backpropagation answers the next question: how should each trainable value change to reduce that loss? In PyTorch, calling `loss.backward()` computes gradients, but it does not update the values; an optimizer or an explicit update does that afterward. section 1.6 section 3.4.2

One value, one gradient

Set `x = torch.tensor(3.0, requires_grad=True)`, then calculate `result = torch.square(x)`. After `result.backward()`, `x.grad` is `tensor(6.)`: the gradient of the result with respect to `x`. If you build a fresh `result` and call `backward()` again without clearing the gradient, `x.grad` becomes `tensor(12.)`. Gradients accumulate, so reset them between steps when accumulation is not intended. section 3.4.1

The mechanism is the chain rule. A model chains simple tensor operations whose derivatives are known. Backpropagation follows their computation graph backward from an output, combining those derivatives to find gradients with respect to earlier tensors. A forward pass produces predictions and loss; the backward pass supplies gradients for the subsequent update. section 2.4.4 section 2.4.3

For a pure-PyTorch training step:

  • Create trainable values such as `W` and `b` with `requires_grad=True`, then compute predictions and a loss. section 3.4.2
  • Call `loss.backward()` and read `W.grad` and `b.grad`. These are gradients of the loss with respect to those values. section 3.4.2
  • Update the values using the gradients. For manual updates, use `torch.no_grad()` so the updates are not included in gradient computation; then set their `.grad` fields to `None`. Alternatively, use `optimizer.step()` and `model.zero_grad()`. section 3.4.2
def training_step(inputs, targets):
    predictions = model(inputs)
    loss = mean_squared_error(targets, predictions)
    loss.backward()
    optimizer.step()
    model.zero_grad()
    return loss

This compact training step shows the practical order: compute loss, call `backward()`, let the optimizer update parameters, then clear gradients. section 3.4.2

The gradient need not be with respect to a model weight. For filter visualization, `loss.backward()` supplies an input image’s gradient for gradient ascent. For a top-class visualization, `.backward()` supplies gradients with respect to convolutional-layer output. In each case, read `.grad` from the tensor whose sensitivity you need. section 10.2.2 section 10.3.2

How Deep Learning with Python, Third Edition covers it

How deep it goes
Explains the chain rule and computation graphs, then works through `.backward()` in a training step without deriving every operation’s derivative.
Examples
Uses a squared scalar and a pure-PyTorch linear classifier to show gradient computation and parameter updates.
What it assumes
Introduces PyTorch tensors and parameters and explains the chain rule before applying `.backward()`.
What you can do after
Apply it via PyTorch `.backward()` to read gradients, update trainable values, and inspect input-image sensitivity.
How it connects
Links computation graphs and the chain rule to loss functions, mini-batch gradient descent, and optimizers.
Cover of Deep Learning with Python, Third Edition
liveBookmanning.com
Read Deep Learning with Python, Third Edition on liveBook Get the book at manning.com

Also explained in

Compare how 17 books explain backpropagation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with Python, Third Edition. The section links point to the parts of the book it draws on.