A locally optimal solution is a point where the objective cannot improve by moving to a nearby point. It may still be worse than the global minimum elsewhere. In neural-network training, a nonconvex error surface can leave randomly initialized weights in a local minimum, where no downhill path is immediately available. section 5.3.1 section 5.3.2
Training practices address local minima indirectly:
- Monitor error or a related metric over training time; continued reduction indicates that optimization is still moving in a useful direction. section 5.3.1
- Shuffle training examples between passes so the optimizer follows a changing path rather than repeatedly descending the same static error surface. section 5.3.2
- Avoid overly ordered sampling in recurrent-network training; consistent alphabetic ordering can specialize predictions and trap optimization in local minima. section 8.3.1
Skiing into a divot
Imagine skiing downhill on a landscape of divots and pits. Starting from different random positions can lead to different stopping points. Reaching the bottom of one nearby pit does not prove that the lowest point on the whole landscape has been found. section 5.3.1 section 5.3.2
Figure 5.7 Nonconvex error curve
Inspect the nonconvex error curve to visualize how optimization can descend into a local minimum instead of reaching the global minimum. section 5.3.1
