A gradient records how the loss changes as a parameter changes. For a parameter θ, ∂L/∂θ gives the direction and rate of that change. Together, these derivatives show how sensitive the loss is to the network’s weights and biases. section 11.2.4
In neural-network training, gradients serve two distinct roles:
- Backpropagation uses the chain rule to calculate gradients by tracing the loss backward through the network. It computes sensitivities but does not change parameters. section 11.2.4
- Gradient descent uses those gradients to update weights and biases. The learning rate controls the size of each adjustment. section 11.2.5
A backward-pass calculation
For a predicted probability of 0.68 and an actual outcome of 1, the output error signal is −0.32. Multiplying it by a hidden-node output of 0.75 gives an output-weight gradient of −0.24. These values describe loss sensitivity; they are not parameter updates. section 11.3.4
The update rule subtracts the learning-rate-scaled gradient from the current parameter; the gradient informs the direction, while the learning rate sets the adjustment size. section 11.2.5


























