Weight decay addresses the tension between fitting training observations and generalizing to new ones by adding a penalty for large weights to the loss. This discourages reliance on a few extreme connections without reducing the network’s layers, nodes, or parameters. section 11.6.2
The tradeoff accepts some limitation in fitting the training data in exchange for expected reliability on new observations. Its role parallels soft-margin SVM regularization; dropout, early stopping, and data augmentation constrain capacity through different mechanisms. section 11.6.2 section 11.8










