How deeply does Deep Learning with R, Third Edition cover recurrent dropout?

Explains mask timing, error propagation, Keras arguments, and a forecasting result without deriving the mechanism mathematically.

Recurrent dropout

As explained in Deep Learning with R, Third Edition

A time-constant dropout mask is a dropout mask whose pattern remains fixed across the timesteps of a sequence.

Recurrent dropout regularizes sequence models by applying one fixed dropout pattern across every timestep. A changing pattern would disrupt the error signal flowing through time, while a time-constant mask lets the network learn across the sequence. section 13.3.4

Dropout-regularized LSTM

For temperature forecasting, configure recurrent dropout inside the LSTM and ordinary dropout after it: layer_lstm(32, recurrent_dropout = 0.25) |> layer_dropout(0.5) |> layer_dense(1). The regularized model can use more units and train longer to reduce overfitting. section 13.3.4

Listing 13.23 Training and evaluating a dropout-regularized LSTM

inputs <- keras_input(shape = c(sequence_length, ncol_input_data))
outputs <- inputs |>
  layer_lstm(32, recurrent_dropout = 0.25) |>
  layer_dropout(0.5) |>  #1
  layer_dense(1)
model <- keras_model(inputs, outputs)

callbacks = list(
  callback_model_checkpoint("jena_lstm_dropout.keras", save_best_only = TRUE)
)

compile(model, optimizer = "adam", loss = "mse", metrics = "mae")history <- model |> fit(
  train_dataset,
  epochs = 50,
  validation_data = val_dataset,
  callbacks = callbacks
)

Inspect the complete R/Keras training and evaluation example, including checkpointing, compilation, metrics, and validation. section 13.3.4

Keras recurrent layers expose separate rates: dropout applies to input units, whereas recurrent_dropout applies to recurrent units and their internal activations. This reduces reliance on shrinking the network for regularization. section 13.3.4 section 13.5

How Deep Learning with R, Third Edition covers it

Cover of Deep Learning with R, Third Edition

liveBook manning.com

Read Deep Learning with R, Third Edition on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Deep Learning with Python, Third EditionA dropout variant for recurrent networks that applies a temporally consistent mask to recurrent computations.

Compare how 2 books explain recurrent dropout →

Related concepts

overfittingaddresses · Overfitting is a modeling condition in which a model fits patterns or idiosyncrasies in its training data too closely and consequently generalizes poorly to new data.
dropoutkind of · Dropout is a neural-network regularization technique that randomly suppresses units, connections, or activations during training to reduce overfitting.
dropout maskhas part · A dropout mask is a pattern specifying which values, units, or contributions are randomly retained or suppressed during training.
Kerasimplemented by · Keras is an open-source, high-level deep-learning API for building, training, evaluating, and deploying neural-network models.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with R, Third Edition. The section links point to the parts of the book it draws on.