How deeply does Math and Architectures of Deep Learning cover momentum?

Explains weighted gradient accumulation, geometric intuition, overshooting, Nesterov lookahead, and Adam’s related state vectors.

Momentum

As explained in Math and Architectures of Deep Learning

An optimization strategy that accelerates parameter updates using accumulated past gradients or a lookahead variant to smooth descent on the loss surface.

Momentum-based gradient descent updates parameters using a weighted average of the current gradient estimate and earlier gradient estimates. Recent gradients receive greater weight, while older gradients receive less. Consistent downhill components reinforce one another, and noisy components tend to cancel, producing a more reliable direction toward the minimum. section 9.4

Momentum connects naturally to several optimization methods:

Momentum amplifies the downhill component progressively as iterations continue. section 9.2.9
Nesterov accelerated gradients use one-step lookahead: estimate the destination, evaluate the gradient there, and average that gradient with the previous update. section 9.2.6
Adam combines momentum’s weighted gradient averaging with RMSProp’s adaptive treatment of dimensions having large or small gradients, adding bias correction for early iterations. section 9.2.9

Overshooting near a minimum

Imagine descending a cone-shaped loss surface. Momentum can carry an update past the minimum, causing the loss to rise. Nesterov evaluates the gradient at the estimated destination; near the minimum, that gradient points opposite to the previous update, so the weighted directions partially cancel and produce a smaller step. section 9.2.6

How Math and Architectures of Deep Learning covers it

Cover of Math and Architectures of Deep Learning

liveBook manning.com

Read Math and Architectures of Deep Learning on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Inside Deep Learning: Math, Algorithms, ModelsA stochastic gradient-based optimization method that uses an accumulated velocity term for parameter updates.
liveBook manning.com

Sutskever's ListMomentum is an optimization technique that incorporates a running history of parameter updates to smooth and accelerate training.
liveBook manning.com

Natural Language Processing in Action, Second EditionMomentum is an optimization hyperparameter that incorporates prior update direction to accelerate or damp parameter movement during gradient-based optimization.
liveBook manning.com

Deep Learning with R, Third EditionMomentum is an optimization technique that incorporates previous parameter updates into the current update to improve movement through the optimization landscape.
liveBook manning.com

Machine Learning Algorithms in DepthMomentum is an optimization technique that incorporates an exponentially weighted moving average of past gradients into parameter updates.

Compare how 6 books explain momentum →

Related concepts

adamused by · Adam is an adaptive stochastic-gradient optimization algorithm that updates model parameters using moving estimates of the first and second moments of gradients.
inner productuses · An inner product is a scalar-valued operation on two vectors that combines their corresponding components, summing their products and, for complex vectors, using conjugation.
stochastic gradient descentconfigures · Stochastic gradient descent is an iterative optimization method that updates parameters using gradients estimated from individual examples or randomly selected small batches.
partial derivativeequivalent to · A partial derivative is the derivative of a multivariable function with respect to one variable while holding the other variables constant.
batch sizeuses · Batch size is the number of items grouped together for processing or computation as a single batch.
Nesterov momentumincludes · Nesterov momentum is an optimization method that evaluates the gradient at a momentum-based lookahead position before updating model parameters.

This explainer was generated by Manning's knowledge graph from the text of Math and Architectures of Deep Learning. The section links point to the parts of the book it draws on.