AdamW
AdamW is an adaptive gradient-based optimization algorithm that combines Adam’s moment-based parameter updates with weight decay applied independently of those updates.
Inside Deep Learning: Math, Algorithms, Models
A variant of Adam that corrects the original algorithm’s issue and is used as the default optimizer in this book.
- How it is used
- Uses AdamW as a practical default optimizer while postponing detailed optimizer tuning until architecture selection is complete.
- How deep it goes
- Defines AdamW, demonstrates PyTorch training, and compares learning curves, but defers the variants’ detailed pros and cons.
- Examples
- Demonstrates fully connected-network training, accuracy plots, cosine annealing, batch-size comparisons, deeper networks, and RNN experiments.
- What you can do after
- Apply it via PyTorch, compare it with SGD variants, and evaluate its behavior with cosine annealing and different architectures.
Read how Inside Deep Learning: Math, Algorithms, Models explains AdamW →
Build a Text-to-Image Generator (from Scratch)
AdamW is an adaptive gradient-based optimizer that decouples weight decay from the parameter update.
- How it is used
- Optimizes ViT classifiers and diffusion denoising U-Nets during gradient-based training.
- How deep it goes
- Defines decoupled weight decay and demonstrates configurations, but stops short of deriving AdamW’s update equations.
- Examples
- Uses PyTorch configurations for CIFAR-10 ViT classification and diffusion-based denoising U-Net training.
- What you can do after
- Apply it via PyTorch to optimize neural networks with configurable learning rates, weight decay, and optimizer parameters.
Read how Build a Text-to-Image Generator (from Scratch) explains AdamW →
Build a Large Language Model (From Scratch)
An adaptive optimization algorithm that maintains historical information to adjust learning rates for neural-network parameters.
- How it is used
- Regularizes GPT pretraining while updating model parameters batch by batch.
- How deep it goes
- Defines the regularization rationale and shows configuration, but stops short of deriving AdamW update equations.
- Examples
- Uses a PyTorch GPTModel configuration with learning rate 0.0004, weight decay 0.1, and ten training epochs.
- What you can do after
- Operate/configure AdamW in a PyTorch training loop and monitor losses for generalization problems.
Read how Build a Large Language Model (From Scratch) explains AdamW →
Related concepts
- adamalternative to · Adam is an adaptive stochastic-gradient optimization algorithm that updates model parameters using moving estimates of the first and second moments of gradients.
- learning rateconfigured by · A learning rate is a step-size parameter that controls the magnitude of updates made by an iterative learning or optimization process.
- optimizationkind of · Optimization is the systematic selection or adjustment of values, actions, or system designs to improve a specified objective, often subject to constraints.
- weight decayuses · Weight decay is a model-training regularization technique that discourages large parameter values by penalizing their magnitudes or shrinking them during optimization.
- optimizer statealternative to · Optimizer state is the auxiliary information an optimization algorithm maintains to determine parameter updates, such as momentum buffers or accumulated statistics.
- pretrainingused by · Pretraining is the initial training of a model on broad data to learn general patterns or capabilities before adaptation to specific tasks or domains.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.