A global optimum is the best attainable point for an objective: its value is no greater than the value at every other point in the domain. In minimization, it is the global minimum, where loss or error is reduced as far as the optimization landscape allows. section 8.2.6 section 6.2.2
Distinguish global optimization from nearby convergence:
- Gradient descent updates parameters in the direction of steepest descent, seeking a global minimum of the objective. section 8.2.6
- A local minimum can look optimal because every nearby direction increases the objective, even though a deeper minimum exists elsewhere. section 6.2.2
- K-means can stop at a local optimum because random centroid initialization does not guarantee the global optimum; repeated runs help select a better result. section 2.3.1
The mountain landscape
Imagine descending a mountain toward the lowest point. A small hollow may appear to be the bottom, but a nearby rise can conceal a deeper valley. The hollow represents a local minimum; the deepest valley represents the global minimum. A purely local optimization method may stop at the hollow without discovering the global optimum. section 6.2.2
Figure 8.8 The concept of gradient descent. It is the mechanism to minimize the loss function.
Use the gradient-descent figure to connect iterative downward movement with minimizing a loss function and approaching a global minimum. section 8.2.6
