A model can fit its training data so closely that it learns patterns too specific to those examples. Regularization deliberately makes perfect training fit harder, aiming for better validation performance. It can limit how much the model memorizes or encourage smoother, more general patterns. Neither a low training loss nor regularization by itself proves that the model generalizes: you need a reliable evaluation procedure. section 5.4.4 section 5.1.2
Start with model capacity
For IMDb review classification, begin with two Dense layers of 16 units each and compare them with versions using 4 or 512 units per layer. A smaller network has fewer parameters available to memorize training examples; a much larger one has more. The goal is not the smallest network possible: too little capacity can underfit. Try architectures on the validation set, starting small and increasing capacity until improvements in validation loss diminish. section 5.4.4
Listing 5.11 Version of the model with lower capacity
model = keras.Sequential( [ layers.Dense(4, activation="relu"), layers.Dense(4, activation="relu"), layers.Dense(1, activation="sigmoid"), ] ) model.compile( optimizer="rmsprop", loss="binary_crossentropy", metrics=["accuracy"], ) history_smaller_model = model.fit( train_data, train_labels, epochs=20, batch_size=512, validation_split=0.4, )
The four-unit Dense layers provide a concrete lower-capacity alternative to the original 16-unit layers. section 5.4.4
When changing capacity is not enough, two other options appear in the same classifier:
- Add L2 weight regularization with `kernel_regularizer=l2(0.002)` on the Dense layers. This adds a regularization term to training rather than simply removing units. section 5.4.4
- Insert `layers.Dropout(0.5)` after each hidden Dense layer. During training, dropout randomly removes part of a layer’s output; the supplied illustration shows training-time rescaling and unchanged activations at test time. section 5.4.4
Regularization can also be part of a model’s objective. A variational autoencoder combines reconstruction loss, which encourages decoded samples to match their inputs, with a regularization loss. The latter uses Kullback–Leibler divergence to nudge encoder output distributions toward a normal distribution centered at zero, helping form a structured latent space and reducing overfitting. section 17.1.2
As a workflow rule, establish a baseline and a model capable of overfitting before regularizing or downsizing it. Choose changes using validation performance, not training or test performance. Keep the test set separate: repeated hyperparameter tuning can overfit the validation set too. section 20.1.5 section 6.4
