Imagine training both arms at the gym while always relying on your stronger right arm. The weaker arm gets little practice. Dropout applies the same idea to a neural network: it randomly omits some units during training so a few strong pathways cannot dominate. section 10.13
At different training steps, random nodes are removed. Because the missing nodes change, other nodes must learn useful patterns instead of depending on the same small group every time. This makes dropout a technique for reducing overfitting, alongside L1 and L2 regularization. section 10.13
Use dropout by making its probability part of the network design:
- Choose a dropout probability p, which controls the random omission used during training. section 10.16
- Place dropout layers between hidden layers when building the architecture. section 10.18
- Tune p with other hyperparameters, using methods such as grid search. section 10.16
A Keras network with dropout
A circular two-class dataset uses two hidden layers of 128 and 64 nodes, with a dropout probability of 0.2 between them: model.add(Dropout(.2)). The model uses ReLU activations in the hidden layers and softmax at the output. After 100 epochs with batches of 10, the reported training accuracy is 0.9000. section 10.18
Figure 10.18 The dropout process. At different epochs, we pick random nodes to remove from the training to give all the nodes an opportunity to update their weights and not have a few single nodes dominating the training.
Shows how different training epochs remove different random nodes, giving all nodes opportunities to update their weights. section 10.13















