Momentum

As explained in Math and Architectures of Deep Learning

An optimization technique that averages current and past gradients to smooth noisy updates and speed movement through flat regions.

Momentum is an optimization technique for minimizing neural-network loss. It combines the current gradient with the previous update so training follows persistent downhill movement instead of reacting fully to each noisy estimate. section 9.2.4 section 9.0

With positive constants γ and η below 1, the update uses a weighted combination of the current gradient and the preceding update. Unrolling the recurrence shows that momentum accumulates gradients from earlier iterations, while older gradients receive progressively smaller weights through powers of γ. section 9.2.4

Momentum changes gradient-based optimization in several useful ways:

• Reinforces components that consistently point downhill. section 9.2.4
• Cancels or weakens non-downhill components that vary across noisy stochastic estimates. section 9.2.4
• Carries nonzero updates through a small flat region where the current gradient is zero. section 9.2.4
• Produces a weighted sum rather than a normalized weighted average; Adam later addresses this weight-sum issue. section 9.2.4

A ball rolling downhill

Imagine a ball moving over a rough hill. Its path zigzags because individual directions are noisy, but repeated downhill components accumulate, increasing its downward velocity and helping it cross a small flat region. section 9.2.4

Figure 9.8 Momentum. Noisy stochastic gradient estimates at different points on the loss surface (thick solid arrows) are not aligned in direction, but they all have a significant downhill component (thin solid arrows). The non-downhill components thin dashed arrows) point in random directions. Hence, averaging tends to strengthen the downhill component and cancel out the non-downhill components.

Use the vector diagram to see why averaging noisy gradient estimates strengthens shared downhill components and cancels random lateral components. section 9.2.4

How Math and Architectures of Deep Learning covers it

How deep it goes

Defines the recurrence, interprets its geometry, unrolls its history, and notes that the weights are not normalized.

Examples

Uses noisy gradient vectors, a rough downhill ball, and a small zero-gradient flat region as explanatory examples.

What it assumes

Assumes loss minimization and gradient-based updates while building intuition for accumulated updates from those foundations.

What you can do after

Evaluate/understand how gradient history smooths updates and helps optimization continue through noise and local flat regions.

How it connects

Connects gradient descent with stochastic gradients, loss functions, Adam, Nesterov, and neural-network training.

liveBookmanning.com

Read Math and Architectures of Deep Learning on liveBook Get the book at manning.com

Related concepts

• gradient descentcontrasts with · Gradient descent is an iterative optimization method that updates parameters opposite the gradient of an objective function to reduce its value.
• locally optimal solutionaddresses · A locally optimal solution is a feasible point whose objective value is no worse than those of sufficiently nearby points, though a better solution may exist elsewhere.
• gradientaddresses · A gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.
• adamequivalent to · Adam is an adaptive stochastic-gradient optimization algorithm that updates model parameters using moving estimates of the first and second moments of gradients.

This explainer was generated by Manning's knowledge graph from the text of Math and Architectures of Deep Learning. The section links point to the parts of the book it draws on.