Adam

As explained in Machine Learning Algorithms in Depth

Adam is an adaptive optimization method that combines momentum-like first-moment estimates with RMSProp-like second-moment scaling and bias correction.

Adam is an adaptive optimization method for stochastic neural-network training. It combines momentum-like first-moment estimates with RMSProp-like second-moment scaling, producing larger progress in gently sloped directions and more cautious updates where gradients are large. section 10.4

The method smooths the stochastic gradient rather than using the raw gradient, then applies a bias-correction mechanism to its moment estimates. Recommended settings are ε = 1e−8, β1 = 0.9, and β2 = 0.999. section 10.4

Adam connects two optimization ideas:

• Momentum contributes a first-moment estimate, which smooths the direction of the stochastic gradient. section 10.4
• RMSProp contributes exponentially weighted second-moment scaling, emphasizing recent gradient history over distant history. section 10.4
• Bias correction compensates for initialization effects in the moment estimates. section 10.4

LeNet on CIFAR-100

A LeNet network trained on the CIFAR-100 dataset provides an optimizer comparison using Keras/TensorFlow. Adam and Nesterov Momentum experimentally produce the highest validation accuracy among the compared optimizers. section 10.4

Figure 10.13 Cross-entropy loss and validation accuracy of LeNet trained on cifar100 using different optimizers

Compare cross-entropy loss and validation accuracy for LeNet trained on CIFAR-100 with different optimizers. section 10.4

How Machine Learning Algorithms in Depth covers it

How deep it goes

Explains Adam’s component ideas, smoothing, bias correction, and recommended constants without deriving the full update equation.

Examples

Compares Adam on a LeNet trained with CIFAR-100 using Keras/TensorFlow and validation-accuracy plots.

What you can do after

Apply Adam via Keras/TensorFlow and compare its loss and validation accuracy with other optimizers.

How it connects

Connects Adam to momentum, RMSProp, stochastic gradients, bias correction, and Nesterov momentum.

liveBookmanning.com

Read Machine Learning Algorithms in Depth on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsAn adaptive optimization algorithm that combines moment estimates with variance-based gradient normalization.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorch
• 

  liveBookmanning.com
  Bayesian Optimization in ActionAdam is an adaptive stochastic-gradient optimization algorithm that maintains moving estimates of first and second moments of gradients.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionAdam is an adaptive stochastic optimization algorithm that uses estimates of first and second moments of gradients to update parameters.
• 

  liveBookmanning.com
  How Large Language Models WorkAdam is an adaptive stochastic-gradient optimization algorithm that maintains estimates of gradient moments to adjust parameter updates.
• 

  liveBookmanning.com
  Math and Architectures of Deep LearningAn adaptive optimization algorithm that combines momentum-like first-moment estimates with per-parameter learning-rate adaptation.

Compare how 7 books explain adam →

Related concepts

• optimizationkind of · Optimization is the systematic selection or adjustment of values, actions, or system designs to improve a specified objective, often subject to constraints.
• stochastic gradient descentkind of · Stochastic gradient descent is an iterative optimization method that updates parameters using gradients estimated from individual examples or randomly selected small batches.
• momentumuses · Momentum is an optimization technique that incorporates a running history of gradients or parameter updates to smooth and accelerate movement during training.
• AdamWalternative to · AdamW is an adaptive gradient-based optimization algorithm that combines Adam’s moment-based parameter updates with weight decay applied independently of those updates.
• gradient descentimplements · Gradient descent is an iterative optimization method that updates parameters opposite the gradient of an objective function to reduce its value.
• RMSpropuses · RMSprop is an adaptive gradient-based optimization algorithm that scales parameter updates using an exponentially decaying average of recent squared gradients.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning Algorithms in Depth. The section links point to the parts of the book it draws on.