liveBook · Manning

AdamW

AdamW is an adaptive gradient-based optimization algorithm that combines Adam’s moment-based updates with weight decay applied independently of those updates.

How 3 Manning books explain it

Inside Deep Learning: Math, Algorithms, Models

A variant of Adam that corrects the original algorithm’s issue and is used as the default optimizer in this book.

How it is used
Uses AdamW as the default optimizer and compares its training behavior with SGD alternatives.
How deep it goes
Defines AdamW as correcting an error in original Adam and compares training behavior, but does not explain the correction’s mechanics.
Examples
Trains a fully connected network with AdamW, plots accuracy, and compares cosine annealing across batch sizes and network depths.
What you can do after
Apply PyTorch’s `torch.optim.AdamW` to train networks and compare results with SGD and scheduled learning rates.

Read how Inside Deep Learning: Math, Algorithms, Models explains AdamW →

Build a Text-to-Image Generator (from Scratch)

AdamW is an adaptive gradient-based optimizer that decouples weight decay from the parameter update.

How it is used
Optimizes ViT classifiers and diffusion denoising U-Nets during gradient-based training.
How deep it goes
Defines decoupled weight decay and demonstrates configurations, but stops short of deriving AdamW’s update equations.
Examples
Uses PyTorch configurations for CIFAR-10 ViT classification and diffusion-based denoising U-Net training.
What you can do after
Apply it via PyTorch to optimize neural networks with configurable learning rates, weight decay, and optimizer parameters.

Read how Build a Text-to-Image Generator (from Scratch) explains AdamW →

Build a Large Language Model (From Scratch)

An adaptive optimization algorithm that maintains historical information to adjust learning rates for neural-network parameters.

How it is used
Regularizes GPT pretraining while updating model parameters batch by batch.
How deep it goes
Defines the regularization rationale and shows configuration, but stops short of deriving AdamW update equations.
Examples
Uses a PyTorch GPTModel configuration with learning rate 0.0004, weight decay 0.1, and ten training epochs.
What you can do after
Operate/configure AdamW in a PyTorch training loop and monitor losses for generalization problems.

Read how Build a Large Language Model (From Scratch) explains AdamW →

Also covered in

What to know first

Related concepts

Questions about AdamW

What is AdamW?

AdamW is an adaptive gradient-based optimization algorithm that combines Adam’s moment-based updates with weight decay applied independently of those updates.

Which Manning books explain AdamW?

Inside Deep Learning: Math, Algorithms, Models, Build a Text-to-Image Generator (from Scratch) and Build a Large Language Model (From Scratch).

What should I know before learning AdamW?

Manning books that teach AdamW usually introduce epoch, parameter, adam and gradient first.

What is the difference between AdamW and adam?

AdamW is an adaptive gradient-based optimization algorithm that combines Adam’s moment-based updates with weight decay applied independently of those updates. Adam is an adaptive stochastic-gradient optimization algorithm that updates parameters using exponential moving estimates of the first and second moments of gradients.

What is AdamW a kind of?

AdamW is a kind of optimization. Optimization is the systematic selection or modification of values, configurations, or implementations to improve a defined objective, often subject to constraints.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.