Recurrent dropout combats overfitting in recurrent networks without forcing regularization to come from a smaller model. Instead of changing which units are dropped at every timestep, it applies one dropout pattern consistently across the sequence. A changing pattern would disrupt the error signal as it propagates through time. section 13.3.4
Listing 13.22 Training and evaluating a dropout-regularized LSTM
inputs = keras.Input(shape=(sequence_length, raw_data.shape[-1])) x = layers.LSTM(32, recurrent_dropout=0.25)(inputs) x = layers.Dropout(0.5)(x) #1 outputs = layers.Dense(1)(x) model = keras.Model(inputs, outputs) callbacks = [ keras.callbacks.ModelCheckpoint( "jena_lstm_dropout.keras", save_best_only=True ) ] model.compile(optimizer="adam", loss="mse", metrics=["mae"]) history = model.fit( train_dataset, epochs=50, validation_data=val_dataset, callbacks=callbacks, )
A practical Keras configuration combines recurrent dropout inside an LSTM with ordinary dropout after the recurrent layer. The example uses recurrent_dropout=0.25, Dropout(0.5), Adam, mean squared error, mean absolute error, validation data, and checkpointing. section 13.3.4
Use recurrent dropout with these rules of thumb:
- Set dropout for input units and recurrent_dropout for recurrent units; recurrent_dropout supplies the temporally consistent recurrent mask. section 13.3.4
- Expect slower convergence after regularization, and compensate with longer training when necessary. section 13.3.4
- On the Jena temperature-forecasting task, a larger LSTM with dropout avoided early overfitting and improved validation and test MAE over the no-learning baseline. section 13.3.4
