A global optimum is the parameter setting with the lowest objective value across the entire domain under consideration. For a neural network, it is the weight vector that produces the smallest possible error for the dataset being optimized. Picture a skier searching for the lowest point on an error surface: a local minimum may be a convenient hollow, while the global minimum is the lowest destination overall. section 5.3.2 section 5.3
Optimization makes the distinction practical:
- Gradient descent follows the steepest downward direction on an error surface, adjusting weights toward lower loss. section 5.3
- A nonconvex surface can trap training at a local minimum, where no immediate downhill path reveals whether a better global minimum exists. section 5.3.1
- Stochastic gradient descent reshuffles training examples, changing the optimization path and helping the model avoid settling too readily in one local minimum. section 5.3.2
- The learning rate limits each step, preventing an individual sample gradient from sending training too far from a useful path. section 5.3.2
- Batch learning evaluates subsets of the data, so each batch supplies a different error surface rather than a complete picture of real-world loss. section 5.3.2
From one example to a dataset
With one input and expected output, plotting error against possible weights produces a minimum: the weights with the least error for that example. Across all training inputs, mean squared error identifies the weight vector that fits the training set as well as possible. In higher dimensions, the same idea applies even though the surface can no longer be pictured as a simple bowl. section 5.3
Figure 5.6 Convex error curve
Use the convex error curve to visualize a bowl-shaped objective and locate its minimum. section 5.3
