Nesterov momentum is an optimization technique that evaluates the gradient after applying the current velocity. Unlike standard momentum, it adds an anticipatory correction to the gradient. section 10.4
Its role in neural-network optimization is defined by two contrasts:
- Momentum accumulates an exponentially decaying moving average of past gradients, so movement depends on gradient magnitude and alignment. section 10.4
- Nesterov momentum evaluates the gradient after applying the current velocity rather than at the current position. section 10.4
- Common momentum parameter values are α = 0.5 and α = 0.9; learning-rate schedules remain part of optimizer configuration. section 10.4
Optimizer comparison
Nesterov momentum appears among the optimizers compared while training a LeNet model on CIFAR-100, alongside loss and validation-accuracy measurements. section 10.4
