Gradient descent updates model parameters to reduce a loss. For parameter θ, it subtracts the gradient scaled by learning rate η; the gradient indicates the local direction of increasing loss, and η sets the step size. section 11.3.5
The update rule applies to each parameter, using its own gradient and the selected learning rate. section 11.3.5
A training cycle links prediction, error measurement, and parameter adjustment:
- Make predictions, then use a loss function to measure their discrepancy from known outcomes. section 11.2.6
- Use backpropagation to calculate gradients, then update parameters from one observation or an averaged mini-batch. section 11.3.5
- Repeat updates across mini-batches and epochs; each step is incremental, and improvements accumulate over training. section 11.2.6
Pedestrian detection
In a worked update, gradients adjust seven network parameters. The predicted probability of a pedestrian rises from 0.68 to 0.69 and cross-entropy loss falls from 0.39 to 0.37, while the braking decision remains unchanged. section 11.3.5 section 11.3.6
Gradient descent follows the local slope of the loss surface but does not guarantee a global minimum. Results can depend on initialization, mini-batch order, optimization method, and learning rate; low training loss alone does not ensure good performance on unseen data. section 11.6.1























