Dropout helps a neural network resist overfitting by randomly setting some layer outputs to zero during training. A dropout rate of 0.5 means that half the output values are dropped at random; rates between 0.2 and 0.5 are common. The aim is to keep the model from relying too heavily on particular activations. section 5.4.4
Add dropout to a review classifier
In an R/Keras IMDb classifier, place `layer_dropout(0.5)` after each of two 16-unit dense layers, before the final sigmoid layer. Fit the model and compare its validation loss with the same classifier without dropout. This tests whether regularization improves performance on held-out reviews, not just training reviews. section 5.4.4
Listing 5.15 Adding dropout to the IMDb model
model <- keras_model_sequential() |> layer_dense(16, activation = "relu") |> layer_dropout(0.5) |> layer_dense(16, activation = "relu") |> layer_dropout(0.5) |> layer_dense(1, activation = "sigmoid") model |> compile( optimizer = "rmsprop", loss = "binary_crossentropy", metrics = "accuracy" ) history_dropout <- model |> fit( train_data, train_labels, epochs = 20, batch_size = 512, validation_split = 0.4 )
The complete R/Keras listing shows where the dropout layers go and how the classifier is compiled and fitted. section 5.4.4
Dropping outputs changes their overall scale. One explanation scales outputs down at test time by the fraction kept. The implementation shown instead divides the surviving outputs by that fraction during training, leaving them unchanged at test time. With a rate of 0.5, surviving training outputs are divided by 0.5. section 5.4.4
Use dropout when a model can fit its training data but needs better generalization. Tune it against validation data, keeping the test set for final evaluation; repeated validation-driven changes can themselves lead to overfitting. Dropout can also complement data augmentation: in an image classifier using both, overfitting began much later than in the original model. In a stacked Transformer decoder, dropout appears within each decoder layer after attention and feedforward blocks. section 6.2.5 section 8.2.5 section 16.2.1
