A neural network learns by changing its weights and biases when its predictions are wrong. A loss function measures that error. Backpropagation computes how the loss changes with each weight; gradient descent uses that information to update the weights repeatedly in the direction that reduces the loss. The practical pairing of backpropagation and gradient descent helped revive interest in multilayer networks in the 1980s. section 2.0
The gradient points toward the steepest local increase in loss, so gradient descent steps the other way. It is a first-order method: it uses the slope at the current position, not information about how the slope bends across the loss surface. That limited view keeps updates computationally efficient, but it can make some terrain hard to navigate. section 2.5.4 section 2.4
A hiker in fog
Imagine estimating a downhill route from a small patch of ground rather than seeing the whole mountain. In mini-batch stochastic gradient descent, a batch of images supplies that local estimate. It is noisy because it is only a sample. Momentum smooths the route using recent steps, while the learning rate controls how far each step goes. section 2.5.4
Long, narrow valleys expose the weakness of slope alone. A large step can bounce across the valley; a small one may creep along its floor. James Martens argued that such curvature, rather than simply an abundance of local minima, helped explain why training deep networks was difficult. His second-order approach used curvature information to traverse these regions and trained autoencoders from random initialization without the layer-wise pretraining previously thought necessary. section 2.4
Figure 2.2 The challenge of optimizing functions shaped like long, narrow valleys. The contour lines represent a valley, with the optimal direction of progress indicated by an arrow running along the valley’s base. Smaller arrows represent steps taken by gradient descent: the left diagram shows large steps oscillating inefficiently across the valley, while the right diagram shows small steps making slow progress along the base. The “pathological” nature does not arise from high or low curvature, but from the combination of high curvature across the valley and low curvature along it. Used with permission granted by the author (James Martens).
The contour diagrams show the trade-off in a narrow valley: larger gradient-descent steps oscillate across it, while smaller steps advance slowly along it. section 2.4
AlexNet nevertheless showed the practical appeal of the simpler method at scale. Its training used mini-batch stochastic gradient descent with 128-image batches and momentum of 0.9. The initial learning rate was 0.01; it was reduced tenfold whenever validation accuracy flattened. The settings illustrate an engineering balance: use inexpensive local updates, then adjust their size as progress slows. section 2.5.4
