A gradient is a vector of partial derivatives: it records how a function changes as its parameters or inputs change. During neural-network training, each component describes how strongly a weight affects the loss, or prediction error. section 5.2.4
Training uses the gradient to decide how weights should move. Backpropagation computes these slopes and communicates adjustments through the network so the loss decreases; the learning rate controls how far each update travels down the slope. section 5.2.4 section 7.5.4
A practical training loop treats the gradient as part of an update cycle:
- Run a forward pass to produce a prediction. section 8.3
- Compare the prediction with the true output to obtain an error or loss. section 5.2.4 section 8.3
- Propagate that error backward and adjust weights according to their gradients. section 5.2.4 section 8.3
- Choose an update distance with a learning rate or optimizer such as Adam or RMSprop. section 7.5.4
Gradients across network types
In a CNN, RMSprop can average recent gradients with an exponentially decaying window to improve parameter-gradient estimates. In an RNN, gradients travel backward through tokens and layers serially; they can disappear or explode, making recurrent training particularly temperamental. section 7.5.4 section 8.7
