Adam
Adam is an adaptive stochastic-gradient optimization algorithm that updates parameters using exponential moving estimates of the first and second moments of gradients.
Inside Deep Learning: Math, Algorithms, Models
An adaptive optimization algorithm that combines moment estimates with variance-based gradient normalization.
- How it is used
- Optimizes neural-network training with adaptive gradient updates, including separate generator and discriminator updates in a GAN.
- How deep it goes
- Explains the gradient moving average and squared-gradient approximation, then shows PyTorch optimizer setup rather than a from-scratch implementation.
- Examples
- Uses AdamW training and accuracy plots alongside SGD variants, plus separate AdamW optimizers in a GAN training loop.
- What you can do after
- Apply it via PyTorch’s torch.optim.AdamW with default settings or configure separate optimizers and task-specific betas for a GAN.
Read how Inside Deep Learning: Math, Algorithms, Models explains adam →
Learn Generative AI with PyTorch
- How it is used
- Uses Adam to update neural-network parameters during supervised and generative-model training.
- How deep it goes
- Explains gradient history, learning rate, beta settings, and training-loop calls without deriving Adam’s update equations.
- Examples
- Uses clothing classification, GAN image generation, handwritten-digit reconstruction, and music generation as training examples.
- What you can do after
- Applies Adam via PyTorch to model parameters, sets learning rates and betas, and performs optimizer steps in a training loop.
Build AI Drug Discovery Pipelines
Adam is an adaptive stochastic-gradient optimization algorithm that maintains running estimates of gradient moments to update model parameters.
- How it is used
- Optimizes neural networks that predict drug-target binding affinities or act as surrogates for docking experiments.
- How deep it goes
- Explains Adam’s training-loop role and configuration, but does not derive its moment estimates or update equations.
- Examples
- Demonstrates Adam in PyTorch code for neural-network training, graph-based affinity prediction, and active-learning surrogate modeling.
- What you can do after
- Apply it via PyTorch to train affinity-prediction models with minibatches and a tunable learning rate.
Fight Fraud with Machine Learning
Adam is an adaptive stochastic optimization algorithm that updates model parameters using estimates of first and second moments of gradients.
- How it is used
- Trains deep neural networks for classification tasks such as fake-voice detection and illicit Bitcoin transaction detection.
- How deep it goes
- Explains Adam’s adaptive rates and momentum, then demonstrates configuration and training loops without deriving its update equations.
- Examples
- Demonstrates Adam with an AudioTransformer and an MLP trained on fake-voice and Elliptic Bitcoin datasets.
- What you can do after
- Apply it via PyTorch by configuring Adam, running backpropagation, and monitoring validation and test metrics.
Bayesian Optimization in Action
Adam is an adaptive stochastic-gradient optimization algorithm that maintains moving estimates of first and second moments of gradients.
- How it is used
- Optimizes Gaussian-process parameters by minimizing negative likelihood during gradient-descent training.
- How deep it goes
- Implements the training loop and convergence checks but does not explain Adam's internal moment estimates.
- Examples
- Uses constant-mean Gaussian processes, full GP parameter training, 500 iterations, and loss-versus-parameter convergence plots.
- What you can do after
- Apply it via PyTorch to train Gaussian-process parameters and evaluate whether optimization has converged.
Deep Learning with R, Third Edition
Adam is an adaptive stochastic optimization algorithm that uses estimates of first and second moments of gradients to update parameters.
- How it is used
- Configures neural-network training by specifying how weight updates reduce the chosen loss.
- How deep it goes
- Identifies Adam’s optimizer role and default usefulness but does not derive its moment estimates or update equations.
- Examples
- Uses MNIST digit classification and binary probability classification as concrete Adam configurations.
- What you can do after
- Apply Adam via Keras to compile and train classification models.
Read how Deep Learning with R, Third Edition explains adam →
How Large Language Models Work
Adam is an adaptive stochastic-gradient optimization algorithm that maintains estimates of gradient moments to adjust parameter updates.
- How it is used
- Optimizes neural-network parameters during large-scale training as an adaptive form of stochastic gradient descent.
- How deep it goes
- Explains momentum, local minima, and memory cost through analogy, but stops short of equations or update rules.
- Examples
- Uses a ball rolling down a loss curve and figures showing parameters, global minima, and local minima.
- What you can do after
- Evaluate Adam’s optimization benefit against its substantially higher training-memory requirement.
Math and Architectures of Deep Learning
An adaptive optimization algorithm that combines momentum-like first-moment estimates with per-parameter learning-rate adaptation.
- How it is used
- Uses combined momentum and adaptive learning rates to minimize neural-network loss during iterative training.
- How deep it goes
- Defines two state vectors, weighted averaging, parameter updates, and bias correction, then stops short of deriving every displayed equation.
- Examples
- Uses state-vector equations and a PyTorch comparison listing with SGD, momentum, AdaGrad, RMSProp, and Adam.
- What you can do after
- Apply it via PyTorch by configuring Adam with parameters, a learning rate, and beta coefficients.
Read how Math and Architectures of Deep Learning explains adam →
Machine Learning Algorithms in Depth
Adam is an adaptive optimization method that combines momentum-like first-moment estimates with RMSProp-like second-moment scaling and bias correction.
- How it is used
- Deploys adaptive stochastic optimization for training neural networks and comparing optimizer performance.
- How deep it goes
- Explains Adam’s component ideas, smoothing, bias correction, and recommended constants without deriving the full update equation.
- Examples
- Compares Adam on a LeNet trained with CIFAR-100 using Keras/TensorFlow and validation-accuracy plots.
- What you can do after
- Apply Adam via Keras/TensorFlow and compare its loss and validation accuracy with other optimizers.
Read how Machine Learning Algorithms in Depth explains adam →
Building Specialized AI Systems
Adam is an adaptive gradient-based optimization algorithm that maintains per-parameter estimates of first and second gradient moments.
- How it is used
- Supports LLM fine-tuning by adapting update sizes independently across model parameters.
- How deep it goes
- Defines per-parameter learning-rate adaptation and contrasts Adam with SGD, AdamW, and Adafactor without detailing optimizer internals.
- Examples
- Uses a hiking-assistant analogy contrasting steep and gentle terrain to illustrate parameter-specific step sizes.
- What you can do after
- Evaluate optimizer choices against update behavior, model scale, memory demands, and overfitting concerns.
Also covered in
- Grokking Machine Learning, Second EditionIn early access (MEAP)
- Deep Learning with PyTorch, Second Edition
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- mean squared errorMean squared error is the arithmetic mean of the squared differences between corresponding predicted and reference numerical values.
- overfittingOverfitting is a modeling condition in which a model adapts too closely to particular data, including its noise or idiosyncrasies, and consequently generalizes poorly to new data.
- objective functionAn objective function is a mathematical function that assigns values to candidate solutions or parameter settings, which an optimization procedure seeks to minimize or maximize.
Related concepts
- optimizationkind of · Optimization is the systematic selection or modification of values, configurations, or implementations to improve a defined objective, often subject to constraints.
- stochastic gradient descentkind of · Stochastic gradient descent is an iterative optimization method that updates parameters using gradients estimated from individual examples or randomly sampled mini-batches.
- momentumuses · Momentum is an optimization technique that incorporates a running history of gradients or parameter updates to smooth and accelerate movement during training.
- AdamWalternative to · AdamW is an adaptive gradient-based optimization algorithm that combines Adam’s moment-based updates with weight decay applied independently of those updates.
- gradient descentimplements · Gradient descent is an iterative optimization method that updates parameters in the direction opposite an objective function’s gradient to reduce the objective’s value.
Questions about adam
What is adam?
Adam is an adaptive stochastic-gradient optimization algorithm that updates parameters using exponential moving estimates of the first and second moments of gradients.
Which Manning books explain adam?
Inside Deep Learning: Math, Algorithms, Models, Learn Generative AI with PyTorch, Build AI Drug Discovery Pipelines, Fight Fraud with Machine Learning, Bayesian Optimization in Action, Deep Learning with R, Third Edition, How Large Language Models Work, Math and Architectures of Deep Learning and 2 more.
What should I know before learning adam?
Manning books that teach adam usually introduce language model, mean squared error, overfitting and objective function first.
What is the difference between adam and AdamW?
Adam is an adaptive stochastic-gradient optimization algorithm that updates parameters using exponential moving estimates of the first and second moments of gradients. AdamW is an adaptive gradient-based optimization algorithm that combines Adam’s moment-based updates with weight decay applied independently of those updates.
What is adam a kind of?
Adam is a kind of optimization. Optimization is the systematic selection or modification of values, configurations, or implementations to improve a defined objective, often subject to constraints.
What is adam also called?
Adam is also called adaptive moment estimation.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.









