liveBook · Manning

Weight decay

As explained in Timeless Algorithms, The Foundational Ideas

Weight decay is a regularization technique that penalizes large weights during model training.

Weight decay addresses the tension between fitting training observations and generalizing to new ones by adding a penalty for large weights to the loss. This discourages reliance on a few extreme connections without reducing the network’s layers, nodes, or parameters. section 11.6.2

The tradeoff accepts some limitation in fitting the training data in exchange for expected reliability on new observations. Its role parallels soft-margin SVM regularization; dropout, early stopping, and data augmentation constrain capacity through different mechanisms. section 11.6.2 section 11.8

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Defines the loss penalty and explains its effect on reliance on large weights and generalization.
What you can do after
Understand how penalizing large weights trades some training fit for expected generalization to new observations.
How it connects
Connects weight decay to soft-margin SVM regularization and distinguishes it from dropout, early stopping, and data augmentation.

Also explained in

Compare how 11 books explain weight decay →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.