liveBook · Manning

Gradient descent

As explained in Grokking Machine Learning, Second Edition

Gradient descent is an iterative optimization technique that repeatedly moves parameters in a direction that decreases an objective function.

Gradient descent is a way to improve a model by repeatedly taking small steps that reduce its error. Imagine standing on a foggy mountain: look nearby, choose the steepest downward direction, take one step, and repeat. Here, each point on the mountain represents a model, and its height represents the model’s error. section 3.24 section 5.16

For linear regression, the error function measures how far a line is from the data. Gradient descent uses the gradient of that function to decide how to adjust the line’s weights and bias. Repeating these adjustments usually finds a model with very low error, although it may stop near rather than exactly at the minimum. section 3.10 section 3.24

The amount of data used for each adjustment gives three common forms:

  • Stochastic gradient descent uses one data point at a time. section 3.26
  • Mini-batch gradient descent uses a small group of data points, which can make large datasets more manageable. section 3.26
  • Batch gradient descent uses the entire dataset for each update. section 3.26

From fitting lines to separating points

A linear regression model starts with a line and moves it toward a better fit by reducing absolute or squared error. A perceptron uses the same small-step idea, but moves a line toward separating two groups of points instead of fitting one line closely to all points. section 3.20 section 5.16

Figure 3.20 We are on top of Mount Errorest and wish to get to the bottom, but we can’t see very far. A way to go down is to look at all the directions in which we can take one step and figure out which one helps us descend the most. Then we are one step closer to the bottom.

Use the mountain illustration to follow the repeated choice of a downward direction and the small step toward lower error. section 3.24

How Grokking Machine Learning, Second Edition covers it

How deep it goes
Explains the descent intuition, error functions, gradients, and data-batch variants, while deferring full calculus derivations.
Examples
Uses a mountain descent, fitted lines, absolute or squared error, and separating lines for two point groups.
What it assumes
Assumes basic familiarity with model parameters and error, while building the optimization intuition from a mountain analogy.
What you can do after
Apply gradient descent to reduce linear-model error using one point, a mini-batch, or the full dataset per iteration.
How it connects
Connects gradient descent with linear regression, error functions, gradients, mini-batch learning, and perceptron training.

Also explained in

Compare how 24 books explain gradient descent →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning, Second Edition. The section links point to the parts of the book it draws on.