Momentum
Momentum is an optimization technique that incorporates a running history of gradients or parameter updates to smooth and accelerate movement during training.
Inside Deep Learning: Math, Algorithms, Models
A stochastic gradient-based optimization method that uses an accumulated velocity term for parameter updates.
- How it is used
- Improves SGD updates by building velocity in consistent gradient directions.
- How deep it goes
- Explains velocity accumulation, shrinking contributions from older gradients, overshoot, and a Nesterov comparison.
- Examples
- Uses oscillating optimization sketches and PyTorch SGD configurations with momentum 0.9.
- What you can do after
- Apply momentum through the momentum and nesterov options in PyTorch torch.optim.SGD.
Read how Inside Deep Learning: Math, Algorithms, Models explains momentum →
Sutskever's List
Momentum is an optimization technique that incorporates a running history of parameter updates to smooth and accelerate training.
- How it is used
- Stabilizes and accelerates mini-batch SGD updates during neural-network training.
- How deep it goes
- Explains momentum through noisy steps and a hiking analogy, but gives no mathematical update rule.
- Examples
- Uses a fog-bound hiker to illustrate noisy gradient estimates and momentum's smoothing effect.
- What you can do after
- Apply momentum to smooth noisy mini-batch updates and move through shallow plateaus during neural-network training.
Math and Architectures of Deep Learning
An optimization strategy that accelerates parameter updates using accumulated past gradients or a lookahead variant to smooth descent on the loss surface.
- How it is used
- Uses accumulated gradient direction to make optimization moves more reliable during loss minimization.
- How deep it goes
- Explains weighted gradient accumulation, geometric intuition, overshooting, Nesterov lookahead, and Adam’s related state vectors.
- Examples
- Uses a cone-shaped loss surface, overshooting diagrams, and PyTorch optimizer configurations as concrete illustrations.
- What you can do after
- Apply it via PyTorch by configuring SGD with momentum or Nesterov momentum and comparing related optimizers.
Read how Math and Architectures of Deep Learning explains momentum →
Natural Language Processing in Action, Second Edition
Momentum is an optimization hyperparameter that incorporates prior update direction to accelerate or damp parameter movement during gradient-based optimization.
- How it is used
- Uses momentum to control parameter movement during stochastic gradient descent for a neural-network classifier.
- How deep it goes
- Defines acceleration and damping behavior, then demonstrates configuration; stops short of deriving the update equation or systematic tuning procedure.
- Examples
- Demonstrates momentum in a PyTorch single-neuron classifier trained for 200 epochs on vectorized name data.
- What you can do after
- Apply momentum via PyTorch's SGD optimizer and inspect training loss and test accuracy across epochs.
Read how Natural Language Processing in Action, Second Edition explains momentum →
Deep Learning with R, Third Edition
Momentum is an optimization technique that incorporates previous parameter updates into the current update to improve movement through the optimization landscape.
- How it is used
- Improves stochastic gradient descent by accelerating convergence and helping optimization escape local minima.
- How deep it goes
- Explains the velocity intuition and gives a naive parameter-update implementation, but does not cover tuning or framework-specific configuration.
- Examples
- Uses one-dimensional loss curves, a rolling-ball analogy, and pseudocode that tracks past velocity.
- What you can do after
- Build it from scratch by retaining the previous velocity and combining it with each new gradient update.
Read how Deep Learning with R, Third Edition explains momentum →
Machine Learning Algorithms in Depth
Momentum is an optimization technique that incorporates an exponentially weighted moving average of past gradients into parameter updates.
- How it is used
- Uses accumulated gradient direction to improve neural-network parameter updates during optimization.
- How deep it goes
- Explains exponential averaging, gradient alignment, parameter choice, and Nesterov’s evaluation difference, but does not provide a full derivation.
- Examples
- Compares optimizers through LeNet training on CIFAR-100 using cross-entropy loss and validation accuracy.
- What you can do after
- Evaluate momentum alongside alternative optimizers using training loss and validation accuracy.
Read how Machine Learning Algorithms in Depth explains momentum →
Also covered in
- Data Without Labels
- Machines that ThinkIn early access (MEAP)
- Advanced Algorithms and Data Structures
What to know first
- objective functionAn objective function is a mathematical function that assigns values to candidate solutions or parameter settings, which an optimization procedure seeks to minimize or maximize.
- gradientA gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.
- accuracyAccuracy is the degree to which a result agrees with a specified truth, reference, or criterion, commonly measured as the proportion of evaluated predictions that are correct.
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
Related concepts
- adamused by · Adam is an adaptive stochastic-gradient optimization algorithm that updates parameters using exponential moving estimates of the first and second moments of gradients.
- inner productuses · An inner product is a scalar-valued operation on two vectors that generalizes the dot product, which for real coordinate vectors is the sum of products of corresponding components.
- partial derivativeequivalent to · A partial derivative is the derivative of a multivariable function with respect to one variable while the other variables are held constant.
- stochastic gradient descentconfigures · Stochastic gradient descent is an iterative optimization method that updates parameters using gradients estimated from individual examples or randomly sampled mini-batches.
- AdaGradused by · AdaGrad is an adaptive optimization algorithm that adjusts each parameter’s learning rate using the historical accumulation of its squared gradients.
Questions about momentum
What is momentum?
Momentum is an optimization technique that incorporates a running history of gradients or parameter updates to smooth and accelerate movement during training.
Which Manning books explain momentum?
Inside Deep Learning: Math, Algorithms, Models, Sutskever's List, Math and Architectures of Deep Learning, Natural Language Processing in Action, Second Edition, Deep Learning with R, Third Edition and Machine Learning Algorithms in Depth.
What should I know before learning momentum?
Manning books that teach momentum usually introduce objective function, gradient, accuracy and language model first.
What is momentum a kind of?
Momentum is a kind of adaptive optimization algorithm. An adaptive optimization algorithm adjusts parameter updates using information from the optimization history or current gradients.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.





