Nesterov momentum
Nesterov momentum is an optimization method that evaluates the gradient at a momentum-based lookahead position before updating model parameters.
Inside Deep Learning: Math, Algorithms, Models
A momentum variant that first applies the accumulated velocity, then computes a gradient at the lookahead position to correct the update sooner.
- How it is used
- Uses lookahead gradients to correct momentum-driven turns sooner during neural-network optimization.
- How deep it goes
- Explains the update sequence and geometric motivation, then stops short of a complete equation derivation.
- Examples
- Uses overshooting momentum diagrams, a lookahead geometry figure, and PyTorch accuracy comparisons.
- What you can do after
- Apply it via PyTorch's SGD optimizer by enabling momentum and the nesterov option.
Read how Inside Deep Learning: Math, Algorithms, Models explains Nesterov momentum →
Math and Architectures of Deep Learning
A momentum-based optimization method that evaluates the gradient at a lookahead point before combining it with past updates.
- How it is used
- Reduces overshooting while optimizing neural-network loss with momentum-based updates.
- How deep it goes
- Explains the lookahead sequence, weighted update, and geometric cancellation, but does not provide the full equation’s terms.
- Examples
- Uses loss-surface overshoot, one-step lookahead, opposite gradients, cancellation, and a cone-like geometric intuition.
- What you can do after
- Evaluate when lookahead gradients can produce safer, smaller optimization steps than pure momentum.
Read how Math and Architectures of Deep Learning explains Nesterov momentum →
Machine Learning Algorithms in Depth
Nesterov momentum is an optimization technique that evaluates the gradient after applying the current velocity to obtain an anticipatory correction.
- How it is used
- Uses an anticipatory gradient correction as a neural-network optimizer alongside stochastic gradient descent and momentum.
- How deep it goes
- Explains the evaluation-point distinction and correction factor, without supplying update equations or a dedicated implementation.
- Examples
- Compares optimizers, including Nesterov momentum, while training LeNet on CIFAR-100.
- What you can do after
- Evaluate its optimizer role by distinguishing anticipatory gradient evaluation from standard momentum.
Read how Machine Learning Algorithms in Depth explains Nesterov momentum →
Related concepts
- momentumkind of · Momentum is an optimization technique that incorporates a running history of gradients or parameter updates to smooth and accelerate movement during training.
- gradientuses · A gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.
- weighted averageuses · A weighted average is an average in which each value contributes according to an assigned numerical weight.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.