A first-order method updates model weights using the local slope of the loss, without using curvature to adjust its direction or step size. Gradient descent moves opposite the gradient, toward lower loss. Its simplicity makes it computationally efficient, but in long, narrow valleys it can oscillate across the valley or take tiny steps along it. section 2.4 section 2.5.4
James Martens argued that difficult neural-network training could reflect the limits of first-order methods, rather than an excess of local minima. His second-order approach used curvature to navigate narrow valleys and trained autoencoders from scratch, challenging the belief that they required layer-wise pretraining. section 2.4
Mini-batch SGD and the hiker
A hiker in fog samples nearby ground to estimate which way slopes downhill, then takes a step. A mini-batch provides a similarly local, noisy gradient estimate; momentum averages recent steps to reduce jitter. AlexNet used mini-batch SGD with momentum 0.9, batches of 128 images, and a learning rate reduced tenfold when validation accuracy flattened. section 2.5.4
Figure 2.2 The challenge of optimizing functions shaped like long, narrow valleys. The contour lines represent a valley, with the optimal direction of progress indicated by an arrow running along the valley’s base. Smaller arrows represent steps taken by gradient descent: the left diagram shows large steps oscillating inefficiently across the valley, while the right diagram shows small steps making slow progress along the base. The “pathological” nature does not arise from high or low curvature, but from the combination of high curvature across the valley and low curvature along it. Used with permission granted by the author (James Martens).
Compare large gradient-descent steps that oscillate across a narrow valley with smaller steps that advance slowly along its base. section 2.4
