A network can fit its training examples so closely that it performs worse on new data. Regularization deliberately limits that fit: it reduces what the model can memorize or encourages smoother, more general patterns. The aim is better validation performance, not a perfect training score. section 5.1.2 section 5.4.4
Add an L2 penalty to an IMDb classifier
In R/Keras, attach an L2 regularizer to each hidden dense layer, then fit the model with a validation split: [train_data, train_labels] <- dataset_imdb(num_words = 10000)[[1]] vectorize <- function(sequences) { results <- matrix(0, nrow = length(sequences), ncol = 10000) for (i in seq_along(sequences)) { idx <- sequences[[i]] + 1L results[i, idx[idx <= 10000]] <- 1 } results } train_data <- vectorize(train_data) model <- keras_model_sequential() |> layer_dense(16, activation = "relu", kernel_regularizer = regularizer_l2(0.002)) |> layer_dense(16, activation = "relu", kernel_regularizer = regularizer_l2(0.002)) |> layer_dense(1, activation = "sigmoid") model |> compile(optimizer = "rmsprop", loss = "binary_crossentropy", metrics = "accuracy") history <- model |> fit(train_data, train_labels, epochs = 20, batch_size = 512, validation_split = 0.4) Compare history$metrics$val_loss with an otherwise identical model without kernel_regularizer. L2 adds a weight penalty to the training loss; the comparison asks whether that constraint helps on held-out reviews. section 5.4.4 section 7.4.5
Other practical ways to constrain the same classifier are:
- Reduce capacity: try fewer layers or units. A smaller model has less room to memorize, but one that is too small may underfit. Compare candidate sizes by validation loss; there is no fixed formula for choosing them. section 5.4.4
- Use dropout: insert layer_dropout(0.5) after hidden layers. It randomly zeros activations during training; compare its validation loss with the unmodified model. section 5.4.4
- Try L1 or combined L1/L2 weight regularization. Early stopping is another option once overfitting begins. section 5.4.4 section 5.5
Listing 5.15 Adding dropout to the IMDb model
model <- keras_model_sequential() |> layer_dense(16, activation = "relu") |> layer_dropout(0.5) |> layer_dense(16, activation = "relu") |> layer_dropout(0.5) |> layer_dense(1, activation = "sigmoid") model |> compile( optimizer = "rmsprop", loss = "binary_crossentropy", metrics = "accuracy" ) history_dropout <- model |> fit( train_data, train_labels, epochs = 20, batch_size = 512, validation_split = 0.4 )
The dropout version of the IMDb classifier shows exactly where to place dropout layers between dense layers. section 5.4.4
First establish a model that can learn and overfit. Then change its architecture or regularization, retrain, and judge each change on validation data—not the test set. Repeated tuning can itself overfit the validation process, so reserve a separate test set for final evaluation. More or better data can also improve generalization. section 6.2.5 section 5.5
Regularization can constrain more than classifier weights. In a variational autoencoder, a regularization loss nudges the encoder’s latent distribution toward a normal distribution, helping make the latent space structured while reducing overfitting. section 17.1.2
