liveBook · Manning

Dropout

As explained in Timeless Algorithms, The Foundational Ideas

Dropout is a neural-network regularization technique that randomly sets selected unit outputs to zero during training.

Dropout regularizes a neural network by temporarily setting randomly selected hidden-node outputs to zero during each training step. The network must learn representations that remain useful without repeatedly relying on the same nodes. This constrains how capacity is used and can reduce dependence on training-specific observations. section 11.6.2

Dropout has a loose conceptual resemblance to random forests, but the mechanisms differ: forests aggregate predictions from separately trained trees, while dropout introduces temporary randomness within one network’s training. The complete network is used for predictions. section 11.6.2

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Defines temporary zeroing of randomly selected hidden-node outputs and explains its representational effect, but gives no implementation details.
What you can do after
Evaluate dropout as a way to discourage repeated reliance on the same hidden nodes during training.
How it connects
Connects dropout to regularization, generalization, random forests, weight decay, early stopping, and data augmentation.

Also explained in

Compare how 16 books explain dropout →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.