How deeply does Deep Learning with Python, Third Edition cover weight decay?

Defines L1 and L2 penalties, equates weight decay with L2, and demonstrates Keras usage without deriving optimizer internals.

Weight decay

As explained in Deep Learning with Python, Third Edition

Weight decay is a model-training regularization technique that discourages large parameter values by penalizing their magnitudes or shrinking them during optimization.

Weight decay helps limit overfitting by discouraging large model weights. It does this by adding a cost for large weights to the loss function. In neural networks, weight decay is mathematically equivalent to L2 regularization; L1 regularization instead penalizes the absolute values of weights. section 5.4.4

A practical starting point is to add L2 regularization when an overfitting model needs better generalization. The example applies it to the kernels of two Dense layers in an IMDb review classifier. section 5.4.4

Listing 5.13 Adding L2 weight regularization to the model

from keras.regularizers import l2

model = keras.Sequential(
    [
        layers.Dense(16, kernel_regularizer=l2(0.002), activation="relu"),
        layers.Dense(16, kernel_regularizer=l2(0.002), activation="relu"),
        layers.Dense(1, activation="sigmoid"),
    ]
)
model.compile(
    optimizer="rmsprop",
    loss="binary_crossentropy",
    metrics=["accuracy"],
)
history_l2_reg = model.fit(
    train_data,
    train_labels,
    epochs=20,
    batch_size=512,
    validation_split=0.4,
)

Keras example adding an L2 regularizer to Dense-layer kernels before compiling and training the model. section 5.4.4

How Deep Learning with Python, Third Edition covers it

Cover of Deep Learning with Python, Third Edition

liveBook manning.com

Read Deep Learning with Python, Third Edition on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Grokking Machine LearningRidge regression is a linear regression technique that uses an L2 coefficient penalty, shrinking coefficients toward zero without usually making them zero.
liveBook manning.com

Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIL2 regularization adds a penalty proportional to the squared values of model parameters to constrain model complexity.
liveBook manning.com

Deep Learning with PyTorch, Second EditionAn optimization method that shrinks parameters in proportion to their current values, generally distinct from L2 regularization for adaptive optimizers.
liveBook manning.com

Ensemble Methods for Machine LearningL2 regularization penalizes the squared magnitude of model parameters or other model complexity measures in an optimization objective.
liveBook manning.com

Deep Learning with R, Third EditionL2 regularization is a regularization method that adds a penalty proportional to the squared values of model parameters.
liveBook manning.com

Machine Learning Algorithms in DepthA regularization method that adds a penalty proportional to the squared magnitude of model parameters to discourage large weights.
liveBook manning.com

Causal AIL2 regularization is a regularization method that penalizes the sum of squared parameter values.
liveBook manning.com

Time Series Forecasting Using Foundation ModelsL2 regularization is a penalty method that discourages large model weights by adding their squared magnitudes to the optimization objective.

Compare how 9 books explain weight decay →

Related concepts

regularizationkind of · Regularization is a modeling technique that constrains or penalizes solutions during fitting or optimization to discourage undesirable complexity or behavior and improve generalization or estimation.
overfittingaddresses · Overfitting is a modeling condition in which a model fits patterns or idiosyncrasies in its training data too closely and consequently generalizes poorly to new data.
ridge regressionused by · Ridge regression is linear regression that applies an L2 penalty to coefficient magnitudes, shrinking estimates toward zero to reduce overfitting.
l2 normuses · The L2 norm is the nonnegative Euclidean magnitude of a vector, equal to the square root of the sum of its components’ squares, and to absolute value for a scalar.
AlexNetused by · AlexNet is a convolutional neural network architecture designed for image classification and recognized for its landmark performance in large-scale image recognition.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with Python, Third Edition. The section links point to the parts of the book it draws on.