liveBook · Manning

Gradient descent

As explained in Timeless Algorithms, The Foundational Ideas

Gradient descent is an iterative optimization technique that updates model parameters in the direction that reduces an objective function.

Gradient descent updates model parameters to reduce a loss. For parameter θ, it subtracts the gradient scaled by learning rate η; the gradient indicates the local direction of increasing loss, and η sets the step size. section 11.3.5

θnew=θold-η∂L∂θ

The update rule applies to each parameter, using its own gradient and the selected learning rate. section 11.3.5

A training cycle links prediction, error measurement, and parameter adjustment:

  • Make predictions, then use a loss function to measure their discrepancy from known outcomes. section 11.2.6
  • Use backpropagation to calculate gradients, then update parameters from one observation or an averaged mini-batch. section 11.3.5
  • Repeat updates across mini-batches and epochs; each step is incremental, and improvements accumulate over training. section 11.2.6

Pedestrian detection

In a worked update, gradients adjust seven network parameters. The predicted probability of a pedestrian rises from 0.68 to 0.69 and cross-entropy loss falls from 0.39 to 0.37, while the braking decision remains unchanged. section 11.3.5 section 11.3.6

Gradient descent follows the local slope of the loss surface but does not guarantee a global minimum. Results can depend on initialization, mini-batch order, optimization method, and learning rate; low training loss alone does not ensure good performance on unseen data. section 11.6.1

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Works through a seven-parameter update and mini-batch training, then discusses optimization limits without guaranteeing a global minimum.
Examples
Uses a pedestrian-detection network to show seven parameter updates and a small improvement in predicted probability and loss.
What it assumes
Assumes a loss function and gradients calculated by backpropagation; explains parameter adjustment and the training cycle.
What you can do after
Apply the update rule to parameters using gradients and a selected learning rate, then assess changes in prediction and loss.
How it connects
Connects backpropagation, loss functions, learning rates, mini-batch training, and epochs.

Also explained in

Compare how 24 books explain gradient descent →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.