Second-order methods use curvature—the way a loss function’s slope changes—to choose update directions and step sizes. In 2010, James Martens argued that multilayer networks stalled not because of too many local minima, but because first-order methods such as gradient descent could not navigate pathological curvature. section 2.4
Narrow valleys
In long, narrow valleys, gradient descent may oscillate across the valley or take tiny steps along it. Martens’s curvature-aware approach traversed these regions more efficiently and trained autoencoders from random initialization, without layer-wise pretraining. section 2.4
Figure 2.2 The challenge of optimizing functions shaped like long, narrow valleys. The contour lines represent a valley, with the optimal direction of progress indicated by an arrow running along the valley’s base. Smaller arrows represent steps taken by gradient descent: the left diagram shows large steps oscillating inefficiently across the valley, while the right diagram shows small steps making slow progress along the base. The “pathological” nature does not arise from high or low curvature, but from the combination of high curvature across the valley and low curvature along it. Used with permission granted by the author (James Martens).
Compare gradient descent’s oscillating and slow steps through a narrow valley. section 2.4
