Curvature describes how the slope of a loss function changes across parameter space. Gradient descent follows the slope but is blind to this bending; second-order methods use curvature to adjust step size and direction. section 2.4
James Martens’s 2010 work reframed stalled training as a problem with optimization tools, not an abundance of local minima. In long, narrow valleys, gradient-based methods may oscillate across the valley or take tiny steps along it. A curvature-aware approach trained autoencoders without unsupervised layer-wise pretraining. section 2.4
Figure 2.2 The challenge of optimizing functions shaped like long, narrow valleys. The contour lines represent a valley, with the optimal direction of progress indicated by an arrow running along the valley’s base. Smaller arrows represent steps taken by gradient descent: the left diagram shows large steps oscillating inefficiently across the valley, while the right diagram shows small steps making slow progress along the base. The “pathological” nature does not arise from high or low curvature, but from the combination of high curvature across the valley and low curvature along it. Used with permission granted by the author (James Martens).
Compare gradient descent’s oscillating or slow steps with progress along a valley’s base; the contrast illustrates how curvature affects optimization. section 2.4
