Adam

As explained in Math and Architectures of Deep Learning

An adaptive optimization algorithm that combines momentum-like first-moment estimates with per-parameter learning-rate adaptation.

Adam is an adaptive optimization algorithm that combines momentum-like averaging of gradients with RMSProp-style per-parameter learning-rate adaptation. It updates neural-network weights and biases while minimizing a loss. section 9.2.9 section 9.0

Adam maintains two state vectors during optimization:

• A first state vector forms a weighted average of past gradients, preserving momentum while improving the averaging scheme. section 9.2.9
• A second state vector tracks gradient magnitudes for RMSProp-like adaptation, reducing learning rates along dimensions with large gradients. section 9.2.9
• Bias correction replaces the early, biased state estimates with corrected quantities, improving updates at small iteration counts. section 9.2.9

PyTorch configuration

A typical Adam optimizer is created with a learning rate of 0.001 and beta values of (0.9, 0.999): optim.Adam([params], lr=0.001, betas=(0.9, 0.999)). section 9.2.9

Equation 9.16

Use the equation sequence to inspect Adam's first state-vector update and connect its weighted-average behavior to momentum. section 9.2.9

Listing 9.12 PyTorch code for various optimizers

from torch import optim

sgd_optimizer = optim.SGD([params], lr=0.01)                   #1

sgd_momentum_optimizer = optim.SGD([params], lr=0.01,
                                    momentum=0.9)              #2

sgd_nesterov_momentum_optimizer = optim.SGD([params], lr=0.01, #3
                                    momentum=0.9, nesterov=True)

adagrad_optimizer = optim.Adagrad([params], lr=0.001)

rms_prop_optimizer = RMSprop([params], lr=1e-2,                #4
                       alpha=0.99)

adam_optimizer = optim.Adam([params], lr=0.001,
                       betas=(0.9, 0.999))

Compare Adam's PyTorch construction with implementations of SGD, momentum, Nesterov momentum, AdaGrad, and RMSProp. section 9.2.9

How Math and Architectures of Deep Learning covers it

How deep it goes

Defines two state vectors, weighted averaging, parameter updates, and bias correction, then stops short of deriving every displayed equation.

Examples

Uses state-vector equations and a PyTorch comparison listing with SGD, momentum, AdaGrad, RMSProp, and Adam.

What it assumes

Assumes loss minimization and gradients, while building the comparison from momentum-based gradient descent and RMSProp.

What you can do after

Apply it via PyTorch by configuring Adam with parameters, a learning rate, and beta coefficients.

How it connects

Connects momentum, RMSProp, gradient descent, loss functions, learning rates, and regularization.

liveBookmanning.com

Read Math and Architectures of Deep Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsAn adaptive optimization algorithm that combines moment estimates with variance-based gradient normalization.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorch
• 

  liveBookmanning.com
  Bayesian Optimization in ActionAdam is an adaptive stochastic-gradient optimization algorithm that maintains moving estimates of first and second moments of gradients.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionAdam is an adaptive stochastic optimization algorithm that uses estimates of first and second moments of gradients to update parameters.
• 

  liveBookmanning.com
  How Large Language Models WorkAdam is an adaptive stochastic-gradient optimization algorithm that maintains estimates of gradient moments to adjust parameter updates.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthAdam is an adaptive optimization method that combines momentum-like first-moment estimates with RMSProp-like second-moment scaling and bias correction.

Compare how 7 books explain adam →

Related concepts

• optimizationkind of · Optimization is the systematic selection or adjustment of values, actions, or system designs to improve a specified objective, often subject to constraints.
• stochastic gradient descentkind of · Stochastic gradient descent is an iterative optimization method that updates parameters using gradients estimated from individual examples or randomly selected small batches.
• momentumuses · Momentum is an optimization technique that incorporates a running history of gradients or parameter updates to smooth and accelerate movement during training.
• AdamWalternative to · AdamW is an adaptive gradient-based optimization algorithm that combines Adam’s moment-based parameter updates with weight decay applied independently of those updates.
• gradient descentimplements · Gradient descent is an iterative optimization method that updates parameters opposite the gradient of an objective function to reduce its value.
• RMSpropuses · RMSprop is an adaptive gradient-based optimization algorithm that scales parameter updates using an exponentially decaying average of recent squared gradients.

This explainer was generated by Manning's knowledge graph from the text of Math and Architectures of Deep Learning. The section links point to the parts of the book it draws on.