Backpropagation computes how a loss changes with every trainable weight and bias in a layered neural network. A forward pass first turns an input into a prediction; the backward pass then starts at the output layer and moves toward the input layer, producing gradients that gradient descent uses to update parameters. section 8.6 section 8.4.6
Layering makes this efficient: each layer receives values only from the preceding layer, so the network can be evaluated forward and differentiated backward one layer at a time. Backpropagation avoids deriving one unwieldy closed-form expression for every parameter gradient. section 8.2 section 8.4.5
The key quantity is an auxiliary error variable δ(l): the rate at which loss changes with the pre-activation output of layer l. At the final layer, δ combines prediction error with the activation derivative. For earlier layers, the next layer's error is carried backward through its weights and multiplied by the local activation derivative. This repeated chain-rule calculation yields gradients for each layer's weights and biases. section 8.4.5 section 8.6
Algorithm 8.5 Training a neural network
| Initialize while 𝕃 > threshold do ⊳ Forward pass for l ← 0 to L do end for ⊳ Loss 𝕃 = 1/2 || ⊳ Gradients for the last layer ▽W(L)𝕃 = ▽b(L)𝕃 = ⊳ Gradients for the remaining layers for l ← L – 1 to 0 do ▽W(l)𝕃 = ▽b(l)𝕃 = end for ⊳ Parameter update W = W – r▽W𝕃 b = b – r▽b𝕃 end while |
The training algorithm places backpropagation between loss evaluation and parameter updates: calculate output-layer gradients, propagate gradients through remaining layers, then subtract learning-rate-scaled gradients from weights and biases. section 8.4.6
Calling backward() in PyTorch
In PyTorch, construct a prediction, calculate a scalar loss, and call backward() on that loss. Autograd performs the gradient computation rather than requiring manual δ equations. An optimizer then consumes those gradients: zero old gradients, call mse_loss.backward(), and call optimizer.step(). For a demand-prediction network, repeated iterations reduce MSE as the model's predictions move toward ground-truth demand. section 8.5 section 8.5
A typical training iteration separates distinct jobs:
- Run the forward pass to produce a prediction from the current parameters. section 8.4.6
- Measure prediction error with a loss, such as mean squared error. section 8.5
- Call loss.backward() to compute gradients with respect to model parameters. section 8.5
- Call optimizer.step() to update weights and biases from those gradients. section 8.5
The same mechanism learns convolution weights: layer dimensions and initial values are specified when building the network, while loss.backward() updates the weights behind the scenes. In deep networks, however, repeatedly multiplying small activation derivatives can make early-layer gradients nearly zero. Sigmoid and tanh can cause this vanishing-gradient problem; ReLU is presented as an activation whose positive-side derivative is constant. section 10.6 section 11.2.1
