When a neural network starts fitting its training data better than its validation data, dropout is one way to limit overfitting. During training, it randomly sets some layer outputs to zero. The dropout rate specifies the fraction removed, commonly between 0.2 and 0.5. section 5.4.4 section 13.3.4
One layer output
An output vector [0.2, 0.5, 1.3, 0.8, 1.1] might become [0, 0.5, 1.3, 0, 1.1] during training. With a rate of 0.5, an implementation can double the surviving outputs during training and leave them unchanged at test time. No outputs are randomly dropped at test time. section 5.4.4
Listing 5.15 Adding dropout to the IMDB model
model = keras.Sequential( [ layers.Dense(16, activation="relu"), layers.Dropout(0.5), layers.Dense(16, activation="relu"), layers.Dropout(0.5), layers.Dense(1, activation="sigmoid"), ] ) model.compile( optimizer="rmsprop", loss="binary_crossentropy", metrics=["accuracy"], ) history_dropout = model.fit( train_data, train_labels, epochs=20, batch_size=512, validation_split=0.4, )
The IMDb classification model puts a Keras Dropout(0.5) layer after each of its two hidden Dense layers, providing a direct comparison with the model without dropout. section 5.4.4
Practical decisions:
- Add dropout while tuning for generalization after the model has shown it can fit the training data; compare configurations on validation data, reserving the test set for final evaluation. section 6.2.5
- Consider L1 or L2 weight regularization for a small model. For a large, highly overparameterized model, constraining weights may have less effect, making dropout a useful alternative. section 5.4.4
Recurrent layers need extra care. Applying a different dropout mask at every timestep can disrupt learning; the same mask should persist across timesteps. Keras recurrent layers expose dropout for input units and recurrent_dropout for recurrent units. In a temperature-forecasting example, an LSTM uses recurrent_dropout=0.25, followed by a separate Dropout(0.5) layer. section 13.3.4
