AdamW
AdamW is an adaptive gradient-based optimization algorithm that combines Adam’s moment-based updates with weight decay applied independently of those updates.
Inside Deep Learning: Math, Algorithms, Models
A variant of Adam that corrects the original algorithm’s issue and is used as the default optimizer in this book.
- How it is used
- Uses AdamW as the default optimizer and compares its training behavior with SGD alternatives.
- How deep it goes
- Defines AdamW as correcting an error in original Adam and compares training behavior, but does not explain the correction’s mechanics.
- Examples
- Trains a fully connected network with AdamW, plots accuracy, and compares cosine annealing across batch sizes and network depths.
- What you can do after
- Apply PyTorch’s `torch.optim.AdamW` to train networks and compare results with SGD and scheduled learning rates.
Read how Inside Deep Learning: Math, Algorithms, Models explains AdamW →
Build a Text-to-Image Generator (from Scratch)
AdamW is an adaptive gradient-based optimizer that decouples weight decay from the parameter update.
- How it is used
- Optimizes ViT classifiers and diffusion denoising U-Nets during gradient-based training.
- How deep it goes
- Defines decoupled weight decay and demonstrates configurations, but stops short of deriving AdamW’s update equations.
- Examples
- Uses PyTorch configurations for CIFAR-10 ViT classification and diffusion-based denoising U-Net training.
- What you can do after
- Apply it via PyTorch to optimize neural networks with configurable learning rates, weight decay, and optimizer parameters.
Read how Build a Text-to-Image Generator (from Scratch) explains AdamW →
Build a Large Language Model (From Scratch)
An adaptive optimization algorithm that maintains historical information to adjust learning rates for neural-network parameters.
- How it is used
- Regularizes GPT pretraining while updating model parameters batch by batch.
- How deep it goes
- Defines the regularization rationale and shows configuration, but stops short of deriving AdamW update equations.
- Examples
- Uses a PyTorch GPTModel configuration with learning rate 0.0004, weight decay 0.1, and ten training epochs.
- What you can do after
- Operate/configure AdamW in a PyTorch training loop and monitor losses for generalization problems.
Read how Build a Large Language Model (From Scratch) explains AdamW →
Also covered in
- Learn Generative AI with PyTorch
- Deep Learning with PyTorch, Second Edition
- Deep Learning with R, Third Edition
- Deep Learning with Python, Third Edition
What to know first
- epochAn epoch is one complete pass through the training data during model training.
- parameterA parameter is a value or named input that specifies or influences the behavior, computation, or output of a function, model, or system.
- adamAdam is an adaptive stochastic-gradient optimization algorithm that updates parameters using exponential moving estimates of the first and second moments of gradients.
- gradientA gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.
Related concepts
- adamalternative to · Adam is an adaptive stochastic-gradient optimization algorithm that updates parameters using exponential moving estimates of the first and second moments of gradients.
- learning rateconfigured by · A learning rate is a numerical step-size parameter that controls the magnitude of successive updates to model parameters or estimates during iterative learning or optimization.
- optimizationkind of · Optimization is the systematic selection or modification of values, configurations, or implementations to improve a defined objective, often subject to constraints.
- weight decayuses · A regularization technique that reduces model parameter magnitudes during optimization, either by adding a penalty to the objective or by directly shrinking parameters.
- optimizer statealternative to · Optimizer state is auxiliary information an optimization algorithm maintains across updates to determine or adapt its parameter updates.
- pretrainingused by · Pretraining is the initial training of a model on broad data to learn general patterns or capabilities before adaptation to specific tasks or domains.
Questions about AdamW
What is AdamW?
AdamW is an adaptive gradient-based optimization algorithm that combines Adam’s moment-based updates with weight decay applied independently of those updates.
Which Manning books explain AdamW?
Inside Deep Learning: Math, Algorithms, Models, Build a Text-to-Image Generator (from Scratch) and Build a Large Language Model (From Scratch).
What should I know before learning AdamW?
Manning books that teach AdamW usually introduce epoch, parameter, adam and gradient first.
What is the difference between AdamW and adam?
AdamW is an adaptive gradient-based optimization algorithm that combines Adam’s moment-based updates with weight decay applied independently of those updates. Adam is an adaptive stochastic-gradient optimization algorithm that updates parameters using exponential moving estimates of the first and second moments of gradients.
What is AdamW a kind of?
AdamW is a kind of optimization. Optimization is the systematic selection or modification of values, configurations, or implementations to improve a defined objective, often subject to constraints.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


