How deeply does Build a Large Language Model (From Scratch) cover AdamW?

Defines the regularization rationale and shows configuration, but stops short of deriving AdamW update equations.

AdamW

As explained in Build a Large Language Model (From Scratch)

An adaptive optimization algorithm that maintains historical information to adjust learning rates for neural-network parameters.

AdamW is an Adam variant used to train neural networks. Its weight-decay approach penalizes larger weights, helping reduce model complexity, limit overfitting, and improve generalization. section 5.2

PyTorch configuration

optimizer = torch.optim.AdamW(model.parameters(), lr=0.0004, weight_decay=0.1). Pass this optimizer to the training function, which calls zero_grad(), backpropagation, and step() for each batch.

How Build a Large Language Model (From Scratch) covers it

Cover of Build a Large Language Model (From Scratch)

liveBook manning.com

Read Build a Large Language Model (From Scratch) on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Inside Deep Learning: Math, Algorithms, ModelsA variant of Adam that corrects the original algorithm’s issue and is used as the default optimizer in this book.
liveBook manning.com

Build a Text-to-Image Generator (from Scratch)AdamW is an adaptive gradient-based optimizer that decouples weight decay from the parameter update.

Compare how 3 books explain AdamW →

Related concepts

adamalternative to · Adam is an adaptive stochastic-gradient optimization algorithm that updates model parameters using moving estimates of the first and second moments of gradients.
learning rateconfigured by · A learning rate is a step-size parameter that controls the magnitude of updates made by an iterative learning or optimization process.
optimizationkind of · Optimization is the systematic selection or adjustment of values, actions, or system designs to improve a specified objective, often subject to constraints.
weight decayuses · Weight decay is a model-training regularization technique that discourages large parameter values by penalizing their magnitudes or shrinking them during optimization.
optimizer statealternative to · Optimizer state is the auxiliary information an optimization algorithm maintains to determine parameter updates, such as momentum buffers or accumulated statistics.
pretrainingused by · Pretraining is the initial training of a model on broad data to learn general patterns or capabilities before adaptation to specific tasks or domains.

This explainer was generated by Manning's knowledge graph from the text of Build a Large Language Model (From Scratch). The section links point to the parts of the book it draws on.