Momentum

As explained in Inside Deep Learning: Math, Algorithms, Models

A stochastic gradient-based optimization method that uses an accumulated velocity term for parameter updates.

Momentum improves stochastic gradient descent when gradients repeatedly point in a consistent direction. It accumulates a velocity term so useful directions produce larger steps while older gradients gradually fade. With learning rate η, gradient g_t, and momentum μ ∈ (0,1), update velocity v using the previous velocity and current gradient, then update parameters with Θ_{t+1} = Θ_t − v_{t+1}. section 5.3.1

The implementation follows three practical ideas:

• Reuse prior update history: each new velocity contains a fraction μ of the previous velocity, so repeated directional agreement builds momentum. section 5.3.1
• Decay old information: because μ is below 1, contributions from increasingly old gradients shrink toward zero. section 5.3.1
• Watch for overshooting: a large accumulated velocity can carry parameters past the desired region before the current gradient redirects them. section 5.3.1

PyTorch SGD with momentum

optimizer = torch.optim.SGD(fc_model.parameters(), lr=eta_0, momentum=0.9, nesterov=False) fc_results_momentum = train_network(fc_model, loss_func, train_loader, test_loader=test_loader, epochs=epochs, optimizer=optimizer, score_funcs={'Accuracy': accuracy_score}, device=device) Setting nesterov=True provides a direct comparison with Nesterov momentum. section 5.3.1

Figure 5.12 The naive optimization is almost stuck oscillating between the top-left and bottom-right directions, but it is slowly and consistently making progress in the top-right direction that leads to the solution. If we could build some momentum in that direction, we would not need so many steps.

Visualizes naive optimization oscillating across directions while making slow progress toward the solution, motivating accumulated momentum. section 5.3.1

How Inside Deep Learning: Math, Algorithms, Models covers it

How deep it goes

Derives velocity accumulation and shrinking history, then demonstrates PyTorch configuration and overshooting behavior.

Examples

Uses optimization-landscape figures, PyTorch SGD listings, and accuracy curves comparing SGD, momentum, and Nesterov momentum.

What it assumes

Assumes gradient descent and neural-network training, while reviewing gradients and learning-rate roles before introducing optimizers.

What you can do after

Apply it via PyTorch's SGD optimizer and compare standard momentum with Nesterov momentum.

How it connects

Connects SGD with learning rate schedules, Nesterov momentum, local minima, and hyperparameter optimization.

liveBookmanning.com

Read Inside Deep Learning: Math, Algorithms, Models on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Sutskever's ListMomentum is an optimization technique that incorporates a running history of parameter updates to smooth and accelerate training.
• 

  liveBookmanning.com
  Math and Architectures of Deep LearningAn optimization strategy that accelerates parameter updates using accumulated past gradients or a lookahead variant to smooth descent on the loss surface.
• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionMomentum is an optimization hyperparameter that incorporates prior update direction to accelerate or damp parameter movement during gradient-based optimization.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionMomentum is an optimization technique that incorporates previous parameter updates into the current update to improve movement through the optimization landscape.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthMomentum is an optimization technique that incorporates an exponentially weighted moving average of past gradients into parameter updates.

Compare how 6 books explain momentum →

Related concepts

• adamused by · Adam is an adaptive stochastic-gradient optimization algorithm that updates model parameters using moving estimates of the first and second moments of gradients.
• inner productuses · An inner product is a scalar-valued operation on two vectors that combines their corresponding components, summing their products and, for complex vectors, using conjugation.
• stochastic gradient descentconfigures · Stochastic gradient descent is an iterative optimization method that updates parameters using gradients estimated from individual examples or randomly selected small batches.
• partial derivativeequivalent to · A partial derivative is the derivative of a multivariable function with respect to one variable while holding the other variables constant.
• batch sizeuses · Batch size is the number of items grouped together for processing or computation as a single batch.
• Nesterov momentumincludes · Nesterov momentum is an optimization method that evaluates the gradient at a momentum-based lookahead position before updating model parameters.

This explainer was generated by Manning's knowledge graph from the text of Inside Deep Learning: Math, Algorithms, Models. The section links point to the parts of the book it draws on.