liveBook · Manning

Nesterov momentum

As explained in Inside Deep Learning: Math, Algorithms, Models

A momentum variant that first applies the accumulated velocity, then computes a gradient at the lookahead position to correct the update sooner.

Nesterov momentum improves ordinary momentum by looking ahead. It first moves using accumulated velocity, then evaluates the gradient at that new position, allowing an earlier correction when momentum points the wrong way. section 5.3.1

The update strategy differs from standard momentum in three practical ways:

  • Standard momentum computes the gradient at the current weights before combining it with velocity. section 5.3.1
  • Nesterov momentum first computes a provisional position from the previous velocity, without using new gradient data. section 5.3.1
  • It then evaluates the gradient at the provisional position, which can produce a stronger push back toward the solution. section 5.3.1

PyTorch configuration

Enable the variant through torch.optim.SGD by setting momentum=0.9 and nesterov=True, then compare its test-accuracy curve with SGD and ordinary momentum.

How Inside Deep Learning: Math, Algorithms, Models covers it

How deep it goes
Explains the update sequence and geometric motivation, then stops short of a complete equation derivation.
Examples
Uses overshooting momentum diagrams, a lookahead geometry figure, and PyTorch accuracy comparisons.
What it assumes
Assumes gradients, weights, learning rate, velocity, and ordinary momentum; contrasts the new variant with standard momentum.
What you can do after
Apply it via PyTorch's SGD optimizer by enabling momentum and the nesterov option.
How it connects
Connects SGD, momentum, velocity, gradients, learning rate, and neural-network accuracy.
Cover of Inside Deep Learning: Math, Algorithms, Models
liveBookmanning.com
Read Inside Deep Learning: Math, Algorithms, Models on liveBook Get the book at manning.com

Also explained in

Compare how 3 books explain Nesterov momentum →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Inside Deep Learning: Math, Algorithms, Models. The section links point to the parts of the book it draws on.