Gradient descent is a way to improve a model by repeatedly taking small steps that reduce its error. Imagine standing on a foggy mountain: look nearby, choose the steepest downward direction, take one step, and repeat. Here, each point on the mountain represents a model, and its height represents the model’s error. section 3.24 section 5.16
For linear regression, the error function measures how far a line is from the data. Gradient descent uses the gradient of that function to decide how to adjust the line’s weights and bias. Repeating these adjustments usually finds a model with very low error, although it may stop near rather than exactly at the minimum. section 3.10 section 3.24
The amount of data used for each adjustment gives three common forms:
- Stochastic gradient descent uses one data point at a time. section 3.26
- Mini-batch gradient descent uses a small group of data points, which can make large datasets more manageable. section 3.26
- Batch gradient descent uses the entire dataset for each update. section 3.26
From fitting lines to separating points
A linear regression model starts with a line and moves it toward a better fit by reducing absolute or squared error. A perceptron uses the same small-step idea, but moves a line toward separating two groups of points instead of fitting one line closely to all points. section 3.20 section 5.16
Figure 3.20 We are on top of Mount Errorest and wish to get to the bottom, but we can’t see very far. A way to go down is to look at all the directions in which we can take one step and figure out which one helps us descend the most. Then we are one step closer to the bottom.
Use the mountain illustration to follow the repeated choice of a downward direction and the small step toward lower error. section 3.24























