How deeply does Inside Deep Learning: Math, Algorithms, Models cover adam?

Explains exponential moving averages and squared-gradient variance approximations, but omits a complete update derivation.

Adam

As explained in Inside Deep Learning: Math, Algorithms, Models

An adaptive optimization algorithm that combines moment estimates with variance-based gradient normalization.

Adam, or adaptive moment estimation, extends SGD with momentum by tracking an exponential moving average of gradients and an approximate variance from squared gradients. It gives reliable directions more influence and scales down high-variance directions, often reducing learning-rate tuning. section 5.3.2

The implementation maps the ideas to two running quantities:

Use m instead of momentum velocity v, and β1 instead of the momentum weight μ; multiply the current gradient by (1 − β1) to form an exponential moving average. section 5.3.2
Track squared gradient values as an imperfect variance approximation, using elementwise operations to estimate how consistently each parameter direction behaves. section 5.3.2

GAN-specific Adam settings

GAN training uses separate AdamW optimizers for the generator and discriminator: learning rate 0.0001 with β1 = 0 and β2 = 0.9. Each optimizer updates its own network during alternating discriminator and generator steps. section 9.1.3

How Inside Deep Learning: Math, Algorithms, Models covers it

Cover of Inside Deep Learning: Math, Algorithms, Models

liveBook manning.com

Read Inside Deep Learning: Math, Algorithms, Models on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Learn Generative AI with PyTorch
liveBook manning.com

Bayesian Optimization in ActionAdam is an adaptive stochastic-gradient optimization algorithm that maintains moving estimates of first and second moments of gradients.
liveBook manning.com

Deep Learning with R, Third EditionAdam is an adaptive stochastic optimization algorithm that uses estimates of first and second moments of gradients to update parameters.
liveBook manning.com

How Large Language Models WorkAdam is an adaptive stochastic-gradient optimization algorithm that maintains estimates of gradient moments to adjust parameter updates.
liveBook manning.com

Math and Architectures of Deep LearningAn adaptive optimization algorithm that combines momentum-like first-moment estimates with per-parameter learning-rate adaptation.
liveBook manning.com

Machine Learning Algorithms in DepthAdam is an adaptive optimization method that combines momentum-like first-moment estimates with RMSProp-like second-moment scaling and bias correction.

Compare how 7 books explain adam →

Related concepts

optimizationkind of · Optimization is the systematic selection or adjustment of values, actions, or system designs to improve a specified objective, often subject to constraints.
stochastic gradient descentkind of · Stochastic gradient descent is an iterative optimization method that updates parameters using gradients estimated from individual examples or randomly selected small batches.
momentumuses · Momentum is an optimization technique that incorporates a running history of gradients or parameter updates to smooth and accelerate movement during training.
AdamWalternative to · AdamW is an adaptive gradient-based optimization algorithm that combines Adam’s moment-based parameter updates with weight decay applied independently of those updates.
gradient descentimplements · Gradient descent is an iterative optimization method that updates parameters opposite the gradient of an objective function to reduce its value.
RMSpropuses · RMSprop is an adaptive gradient-based optimization algorithm that scales parameter updates using an exponentially decaying average of recent squared gradients.

This explainer was generated by Manning's knowledge graph from the text of Inside Deep Learning: Math, Algorithms, Models. The section links point to the parts of the book it draws on.