liveBook · Manning

Dropout

As explained in Grokking Machine Learning, Second Edition

A neural-network layer that randomly omits units during training to reduce reliance on specific pathways.

Imagine training both arms at the gym while always relying on your stronger right arm. The weaker arm gets little practice. Dropout applies the same idea to a neural network: it randomly omits some units during training so a few strong pathways cannot dominate. section 10.13

At different training steps, random nodes are removed. Because the missing nodes change, other nodes must learn useful patterns instead of depending on the same small group every time. This makes dropout a technique for reducing overfitting, alongside L1 and L2 regularization. section 10.13

Use dropout by making its probability part of the network design:

  • Choose a dropout probability p, which controls the random omission used during training. section 10.16
  • Place dropout layers between hidden layers when building the architecture. section 10.18
  • Tune p with other hyperparameters, using methods such as grid search. section 10.16

A Keras network with dropout

A circular two-class dataset uses two hidden layers of 128 and 64 nodes, with a dropout probability of 0.2 between them: model.add(Dropout(.2)). The model uses ReLU activations in the hidden layers and softmax at the output. After 100 epochs with batches of 10, the reported training accuracy is 0.9000. section 10.18

Figure 10.18 The dropout process. At different epochs, we pick random nodes to remove from the training to give all the nodes an opportunity to update their weights and not have a few single nodes dominating the training.

Shows how different training epochs remove different random nodes, giving all nodes opportunities to update their weights. section 10.13

How Grokking Machine Learning, Second Edition covers it

How deep it goes
Explains the intuition, random node-removal process, probability hyperparameter, and Keras placement, but not deeper mathematical internals.
Examples
Uses a two-arm gym analogy, a circular two-class dataset, Keras layers, and a two-hidden-layer architecture with probability 0.2.
What it assumes
Assumes basic neural-network layers and training while introducing dropout alongside regularization as an overfitting remedy.
What you can do after
Apply dropout via Keras by inserting Dropout layers and tuning their probability with other neural-network hyperparameters.
How it connects
Links dropout to neural networks, overfitting, regularization, hyperparameters, backpropagation, and Keras.

Also explained in

Compare how 16 books explain dropout →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning, Second Edition. The section links point to the parts of the book it draws on.