How deeply does Math and Architectures of Deep Learning cover Nesterov momentum?

Explains the lookahead sequence, weighted update, and geometric cancellation, but does not provide the full equation’s terms.

Nesterov momentum

As explained in Math and Architectures of Deep Learning

A momentum-based optimization method that evaluates the gradient at a lookahead point before combining it with past updates.

Nesterov momentum is an optimization method designed to reduce momentum’s tendency to overshoot a loss minimum. It estimates the next destination, evaluates the gradient there, and combines that gradient with the previous update. section 9.2.6 section 9.0

The update follows a one-step lookahead:

Estimate the destination by adding the previous step’s update to the current point. section 9.2.6
Compute the gradient at that estimated destination rather than at the current point. section 9.2.6
Take a weighted average of the lookahead gradient and the previous update, then use the result to update weights and biases. section 9.2.6

Why the lookahead helps

Far from the minimum, the lookahead and current gradients are similar, so progress resembles momentum-based gradient descent. Near the minimum, the lookahead gradient can point in the opposite direction from the previous update; partial cancellation produces a smaller step and reduces overshooting. section 9.2.6

How Math and Architectures of Deep Learning covers it

Cover of Math and Architectures of Deep Learning

liveBook manning.com

Read Math and Architectures of Deep Learning on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Inside Deep Learning: Math, Algorithms, ModelsA momentum variant that first applies the accumulated velocity, then computes a gradient at the lookahead position to correct the update sooner.
liveBook manning.com

Machine Learning Algorithms in DepthNesterov momentum is an optimization technique that evaluates the gradient after applying the current velocity to obtain an anticipatory correction.

Compare how 3 books explain Nesterov momentum →

Related concepts

momentumkind of · Momentum is an optimization technique that incorporates a running history of gradients or parameter updates to smooth and accelerate movement during training.
gradientuses · A gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.
weighted averageuses · A weighted average is an average in which each value contributes according to an assigned numerical weight.

This explainer was generated by Manning's knowledge graph from the text of Math and Architectures of Deep Learning. The section links point to the parts of the book it draws on.