Convergence marks the point where repeated optimization updates change so little that the procedure stops, treating the current solution as a local minimizer. section 5.1.1
A tolerance-based convergence check fits into an iterative optimization loop:
- Start with a solution guess and compute an update direction. section 5.1.1
- Choose a step length, update the solution, and compare it with the previous iteration. section 5.1.1
- Stop when the change falls below a specified tolerance; otherwise, repeat from the new solution. section 5.1.1
Branin optimization
Gradient descent starts the Branin search at [-4,-5] and records its optimization path. The steps shrink and may zigzag near a local minimum; different initial guesses can converge to different minima. section 5.1.1
: wold = some initial guess, converged=False
while not converged:
1. compute the direction (d) as negative gradient at wold and normalize
to unit length
2. compute the step length using line search (distance, α)
3. update the solution: wnew = wold + distance *direction = wold + α ⋅ d
4. if change between wnew and wold is below some specified tolerance:
converged=True, so break
5.else set wnew = wold, get ready for the next iteration
Follow the procedural loop from direction calculation through tolerance checking and the next iteration. section 5.1.1
