AdamW

As explained in Inside Deep Learning: Math, Algorithms, Models

A variant of Adam that corrects the original algorithm’s issue and is used as the default optimizer in this book.

AdamW is an Adam variant that fixes an error in the original algorithm. It is used as the default optimizer and reduces the need for extensive learning-rate experimentation. section 5.3.2

In PyTorch, construct it with the model parameters and let its default learning rate stand unless experimentation is necessary. AdamW can train a fully connected network with behavior comparable to momentum-based SGD, while its accuracy curve is somewhat more stable. section 5.3.2

Use the accompanying training figure to compare AdamW with SGD, momentum, and Nesterov momentum. section 5.3.2

Basic PyTorch setup

fc_model.apply(weight_reset) optimizer = torch.optim.AdamW(fc_model.parameters()) fc_results_adam = train_network( fc_model, loss_func, train_loader, test_loader=test_loader, epochs=epochs, optimizer=optimizer, score_funcs={'Accuracy': accuracy_score}, device=device ) section 5.3.2

fc_model.apply(weight_reset) optimizer =
torch.optim.AdamW(fc_model.parameters())                                      #1

scheduler = torch.optim.lr_scheduler. 
 CosineAnnealingLR(optimizer, epochs//3) 
fc_results_adam_coslr = train_network(fc_model, loss_func, train_loader, 
 test_loader=test_loader, epochs=epochs, optimizer=optimizer, 
 lr_schedule=scheduler, score_funcs={’Accuracy’: accuracy_score}, 
 device=device)

fc_model.apply(weight_reset) optimizer =
torch.optim.SGD(fc_model.parameters(), lr=eta_0, momentum=0.9, nesterov=True) #2

scheduler = torch.optim.lr_scheduler. 
 CosineAnnealingLR(optimizer, epochs//3) 
fc_results_nesterov_coslr = train_network(fc_model, loss_func, 
 train_loader, test_loader=test_loader, epochs=epochs, 
 optimizer=optimizer, lr_schedule=scheduler, 
 score_funcs={’Accuracy’: accuracy_score}, device=device)
#1 Adam with cosine annealing
#1 SGD+Nesterov with cosine annealing

Combine AdamW with CosineAnnealingLR to test whether scheduled learning rates improve accuracy. section 5.3.2

How Inside Deep Learning: Math, Algorithms, Models covers it

How deep it goes

Defines AdamW, demonstrates PyTorch training, and compares learning curves, but defers the variants’ detailed pros and cons.

Examples

Demonstrates fully connected-network training, accuracy plots, cosine annealing, batch-size comparisons, deeper networks, and RNN experiments.

What it assumes

Assumes familiarity with Adam, SGD, momentum, neural-network training, and learning-rate schedules.

What you can do after

Apply it via PyTorch, compare it with SGD variants, and evaluate its behavior with cosine annealing and different architectures.

How it connects

Connects AdamW with Adam, SGD, momentum, Nesterov momentum, learning-rate schedules, and cosine annealing.

liveBookmanning.com

Read Inside Deep Learning: Math, Algorithms, Models on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Build a Text-to-Image Generator (from Scratch)AdamW is an adaptive gradient-based optimizer that decouples weight decay from the parameter update.
• 

  liveBookmanning.com
  Build a Large Language Model (From Scratch)An adaptive optimization algorithm that maintains historical information to adjust learning rates for neural-network parameters.

Compare how 3 books explain AdamW →

Related concepts

• adamalternative to · Adam is an adaptive stochastic-gradient optimization algorithm that updates model parameters using moving estimates of the first and second moments of gradients.
• learning rateconfigured by · A learning rate is a step-size parameter that controls the magnitude of updates made by an iterative learning or optimization process.
• optimizationkind of · Optimization is the systematic selection or adjustment of values, actions, or system designs to improve a specified objective, often subject to constraints.
• weight decayuses · Weight decay is a model-training regularization technique that discourages large parameter values by penalizing their magnitudes or shrinking them during optimization.
• optimizer statealternative to · Optimizer state is the auxiliary information an optimization algorithm maintains to determine parameter updates, such as momentum buffers or accumulated statistics.
• pretrainingused by · Pretraining is the initial training of a model on broad data to learn general patterns or capabilities before adaptation to specific tasks or domains.

This explainer was generated by Manning's knowledge graph from the text of Inside Deep Learning: Math, Algorithms, Models. The section links point to the parts of the book it draws on.