A gradient tells a model how its loss changes when its parameters change. For one weight, a partial derivative gives the direction and sensitivity: a positive value means increasing that weight increases error; a negative value means increasing it decreases error. The symbol ∇W collects the partial derivatives for every weight in a layer into a matrix. section 9.26
Gradients turn a mistake into small weight changes:
- Make a prediction and calculate its loss by comparing it with the correct result. section 11.32
- Work backward from the loss. The chain rule links a weight to the error through the layers between them, producing a gradient for each parameter. section 9.26 section 11.30
- Update each weight using its gradient and a learning rate, which controls the size of the change. Repeat with more batches. section 11.30 section 11.32
One small update
Suppose a weight is 0.5, its gradient is −0.5, and the learning rate is 0.01. Subtracting learning rate × gradient gives 0.5 − (0.01 × −0.5) = 0.505. The weight increases slightly because its negative gradient indicates that increasing it should reduce loss. section 9.26 section 11.30
The same idea applies when training different networks. A language model calculates gradients back through its output layers and attention mechanism to its embeddings. A diffusion model uses gradients from noise-prediction loss to update its weights. These are training steps; generating an image with an already trained model is inference. section 11.30 section 12.23
