Weight decay, also called L2 regularization, reduces overfitting by adding a loss penalty proportional to the squared values of the model’s weights. The penalty favors smaller, more regular weight values and discourages unnecessarily complex explanations. It is especially useful after a model has enough capacity to overfit. section 5.4.4 section 5.5
Add L2 regularization to an IMDb classifier
In an R/Keras binary sentiment model, attach the same L2 penalty to the kernels of both hidden layers: model <- keras_model_sequential() |> layer_dense(16, activation = "relu", kernel_regularizer = regularizer_l2(0.002)) |> layer_dense(16, activation = "relu", kernel_regularizer = regularizer_l2(0.002)) |> layer_dense(1, activation = "sigmoid") Compile and train the model as usual, then compare its validation-loss history with the unregularized model. The regularized model is expected to resist the worsening validation performance associated with overfitting. section 5.4.4
Use weight decay as part of a practical regularization workflow:
- First establish that the model can learn and overfit; then use L2 regularization to improve generalization. section 6.2.5
- Tune the regularization strength alongside architecture, learning rate, layer width, dropout, and other model choices. section 6.2.5
- Compare validation behavior without repeatedly treating the test set as a tuning signal. section 6.2.5
- Distinguish L2 from L1 regularization: L2 penalizes squared weights, whereas L1 penalizes their absolute values. section 5.4.4
Listing 5.13 Adding L2 weight regularization to the model
model <- keras_model_sequential() |>
layer_dense(16, activation = "relu",
kernel_regularizer = regularizer_l2(0.002)) |>
layer_dense(16, activation = "relu",
kernel_regularizer = regularizer_l2(0.002)) |>
layer_dense(1, activation = "sigmoid")
model |> compile(
optimizer = "rmsprop",
loss = "binary_crossentropy",
metrics = "accuracy"
)
history_l2_reg <- model |> fit(
train_data, train_labels,
epochs = 20, batch_size = 512, validation_split = 0.4
)
Inspect the complete Keras model definition that applies L2 penalties to its hidden-layer kernels. section 5.4.4
