Recurrent neural network regularization addresses overfitting in sequence models. In 2014, Zaremba, Sutskever, and Vinyals introduced selective dropout tailored to LSTM architectures, a turning point for recurrent networks. section 4.3
Dropout’s basic rationale is to make a model less dependent on particular neurons:
- During training, randomly turning off neurons creates different network variants and encourages robust features; at inference, dropout is disabled. section 4.3
- The recurrent approach selectively adapted dropout to LSTMs; the available description does not specify where dropout is applied within the architecture. section 4.0 section 4.3
Dropout as an ensemble
Randomly disabling neurons produces different hypothetical networks during training. This resembles model ensembling, reducing overfitting without explicitly training separate models. section 4.3
