Momentum

As explained in Deep Learning with R, Third Edition

Momentum is an optimization technique that incorporates previous parameter updates into the current update to improve movement through the optimization landscape.

Momentum improves stochastic gradient descent by combining the current gradient with previous parameter updates. The accumulated movement can increase convergence speed and help the optimization process pass through local minima. section 2.4.3

Momentum is useful because it:

• Uses a physics-inspired velocity analogy: past gradients influence the next parameter movement. section 2.4.3
• Updates parameters from both the current gradient and the previous velocity rather than the current gradient alone. section 2.4.3
• Addresses slow convergence and getting stuck at local minima, although low-dimensional intuitions may not fully represent neural-network training. section 2.4.3

Naive momentum update

A basic implementation stores the previous velocity, combines it with the current gradient, updates the parameter, and then saves the new velocity for the next step. section 2.4.3

Figure 2.19 A local minimum and a global minimum

Compare a local minimum with a global minimum to see why ordinary gradient descent can stop prematurely and why momentum may continue moving. section 2.4.3

How Deep Learning with R, Third Edition covers it

How deep it goes

Explains the velocity intuition and gives a naive parameter-update implementation, but does not cover tuning or framework-specific configuration.

Examples

Uses one-dimensional loss curves, a rolling-ball analogy, and pseudocode that tracks past velocity.

What it assumes

Assumes familiarity with stochastic gradient descent, gradients, parameters, learning rate, and loss landscapes.

What you can do after

Build it from scratch by retaining the previous velocity and combining it with each new gradient update.

How it connects

Connects stochastic gradient descent with optimizers, learning rate, gradients, local minima, and global minima.

liveBookmanning.com

Read Deep Learning with R, Third Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA stochastic gradient-based optimization method that uses an accumulated velocity term for parameter updates.
• 

  liveBookmanning.com
  Sutskever's ListMomentum is an optimization technique that incorporates a running history of parameter updates to smooth and accelerate training.
• 

  liveBookmanning.com
  Math and Architectures of Deep LearningAn optimization strategy that accelerates parameter updates using accumulated past gradients or a lookahead variant to smooth descent on the loss surface.
• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionMomentum is an optimization hyperparameter that incorporates prior update direction to accelerate or damp parameter movement during gradient-based optimization.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthMomentum is an optimization technique that incorporates an exponentially weighted moving average of past gradients into parameter updates.

Compare how 6 books explain momentum →

Related concepts

• adamused by · Adam is an adaptive stochastic-gradient optimization algorithm that updates model parameters using moving estimates of the first and second moments of gradients.
• inner productuses · An inner product is a scalar-valued operation on two vectors that combines their corresponding components, summing their products and, for complex vectors, using conjugation.
• stochastic gradient descentconfigures · Stochastic gradient descent is an iterative optimization method that updates parameters using gradients estimated from individual examples or randomly selected small batches.
• partial derivativeequivalent to · A partial derivative is the derivative of a multivariable function with respect to one variable while holding the other variables constant.
• batch sizeuses · Batch size is the number of items grouped together for processing or computation as a single batch.
• Nesterov momentumincludes · Nesterov momentum is an optimization method that evaluates the gradient at a momentum-based lookahead position before updating model parameters.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with R, Third Edition. The section links point to the parts of the book it draws on.