Adam
Adam is an adaptive stochastic-gradient optimization algorithm that updates model parameters using moving estimates of the first and second moments of gradients.
Inside Deep Learning: Math, Algorithms, Models
An adaptive optimization algorithm that combines moment estimates with variance-based gradient normalization.
- How it is used
- Uses adaptive moment estimates to train neural networks while reducing manual learning-rate tuning.
- How deep it goes
- Explains exponential moving averages and squared-gradient variance approximations, but omits a complete update derivation.
- Examples
- Demonstrates AdamW on a classifier and on a GAN with separate generator and discriminator optimizers.
- What you can do after
- Apply it via PyTorch, selecting defaults for ordinary training or separate low-rate settings for GAN optimization.
Read how Inside Deep Learning: Math, Algorithms, Models explains adam →
Learn Generative AI with PyTorch
- How it is used
- Updates neural-network parameters during classification, image generation, autoencoder reconstruction, and music Transformer training.
- How deep it goes
- Explains gradient history, learning rate, betas, and training-loop calls, but does not derive Adam's update equations.
- Examples
- Uses a binary clothing classifier, separate GAN optimizers, handwritten-digit reconstruction, and a music Transformer.
- What you can do after
- Apply it via PyTorch `torch.optim.Adam` and place its update step after gradient computation in a training loop.
Bayesian Optimization in Action
Adam is an adaptive stochastic-gradient optimization algorithm that maintains moving estimates of first and second moments of gradients.
- How it is used
- Optimizes Gaussian-process parameters by minimizing negative likelihood during gradient-descent training.
- How deep it goes
- Implements the training loop and convergence checks but does not explain Adam's internal moment estimates.
- Examples
- Uses constant-mean Gaussian processes, full GP parameter training, 500 iterations, and loss-versus-parameter convergence plots.
- What you can do after
- Apply it via PyTorch to train Gaussian-process parameters and evaluate whether optimization has converged.
Deep Learning with R, Third Edition
Adam is an adaptive stochastic optimization algorithm that uses estimates of first and second moments of gradients to update parameters.
- How it is used
- Configures neural-network training by specifying how weight updates reduce the chosen loss.
- How deep it goes
- Identifies Adam’s optimizer role and default usefulness but does not derive its moment estimates or update equations.
- Examples
- Uses MNIST digit classification and binary probability classification as concrete Adam configurations.
- What you can do after
- Apply Adam via Keras to compile and train classification models.
Read how Deep Learning with R, Third Edition explains adam →
How Large Language Models Work
Adam is an adaptive stochastic-gradient optimization algorithm that maintains estimates of gradient moments to adjust parameter updates.
- How it is used
- Optimizes neural-network parameters during large-scale training as an adaptive form of stochastic gradient descent.
- How deep it goes
- Explains momentum, local minima, and memory cost through analogy, but stops short of equations or update rules.
- Examples
- Uses a ball rolling down a loss curve and figures showing parameters, global minima, and local minima.
- What you can do after
- Evaluate Adam’s optimization benefit against its substantially higher training-memory requirement.
Math and Architectures of Deep Learning
An adaptive optimization algorithm that combines momentum-like first-moment estimates with per-parameter learning-rate adaptation.
- How it is used
- Uses combined momentum and adaptive learning rates to minimize neural-network loss during iterative training.
- How deep it goes
- Defines two state vectors, weighted averaging, parameter updates, and bias correction, then stops short of deriving every displayed equation.
- Examples
- Uses state-vector equations and a PyTorch comparison listing with SGD, momentum, AdaGrad, RMSProp, and Adam.
- What you can do after
- Apply it via PyTorch by configuring Adam with parameters, a learning rate, and beta coefficients.
Read how Math and Architectures of Deep Learning explains adam →
Machine Learning Algorithms in Depth
Adam is an adaptive optimization method that combines momentum-like first-moment estimates with RMSProp-like second-moment scaling and bias correction.
- How it is used
- Deploys adaptive stochastic optimization for training neural networks and comparing optimizer performance.
- How deep it goes
- Explains Adam’s component ideas, smoothing, bias correction, and recommended constants without deriving the full update equation.
- Examples
- Compares Adam on a LeNet trained with CIFAR-100 using Keras/TensorFlow and validation-accuracy plots.
- What you can do after
- Apply Adam via Keras/TensorFlow and compare its loss and validation accuracy with other optimizers.
Read how Machine Learning Algorithms in Depth explains adam →
Related concepts
- optimizationkind of · Optimization is the systematic selection or adjustment of values, actions, or system designs to improve a specified objective, often subject to constraints.
- stochastic gradient descentkind of · Stochastic gradient descent is an iterative optimization method that updates parameters using gradients estimated from individual examples or randomly selected small batches.
- momentumuses · Momentum is an optimization technique that incorporates a running history of gradients or parameter updates to smooth and accelerate movement during training.
- AdamWalternative to · AdamW is an adaptive gradient-based optimization algorithm that combines Adam’s moment-based parameter updates with weight decay applied independently of those updates.
- gradient descentimplements · Gradient descent is an iterative optimization method that updates parameters opposite the gradient of an objective function to reduce its value.
- RMSpropuses · RMSprop is an adaptive gradient-based optimization algorithm that scales parameter updates using an exponentially decaying average of recent squared gradients.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.