AlexNet’s fully connected layers held most of its parameters, making them especially vulnerable to overfitting: learning the training examples without preserving performance on new ones. Dropout addressed that risk by randomly turning off a fraction of neurons at each training iteration. The changing network discourages neurons from depending on the same tightly coupled partners—the “co-conspirators” in Geoffrey Hinton’s analogy. section 2.5.2
Dropout in AlexNet
At each iteration, AlexNet dropped 50% of the neurons in each of its first two fully connected layers, FC6 and FC7. Dropout was one part of a larger training recipe that also used ReLU activations, weight initialization, momentum, and data augmentation; its particular job was to reduce overfitting. section 2.5.2 section 2.5.5
Figure 2.5 Left: a standard neural network with two hidden layers. Right: an example of a network produced by applying dropout to the network on the left. Used with permission granted by the lead author (Nitish Srivastava).
Compare the standard two-hidden-layer network with one possible network after dropout turns off a subset of its neurons. section 2.5.2
At test time, dropout is disabled. Early implementations reduced layer outputs then to compensate for the neurons active during training. Inverted dropout instead makes the compensation during training, leaving no test-time adjustment; both approaches aim to keep the average signal consistent. Dropout can be understood as an efficient approximation to averaging predictions from many networks, without training each one separately. section 2.5.2 section 4.3
The overfitting problem extended beyond AlexNet. In 2014, Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals introduced selective dropout tailored to LSTM architectures. The engineering lesson was not that one regularizer solved deep learning, but that dropout could be adapted to a particular architecture and combined with other training choices. section 4.3 section 2.5.5
