liveBook · Manning

Dropout

As explained in Grokking Bayes

Dropout is a neural-network regularization technique that randomly omits units or connections during training to reduce co-adaptation and overfitting.

Dropout regularizes a neural network by randomly removing units during training. A removed unit outputs zero for that step, weakening co-adaptation and reducing overfitting so the network can generalize better. section 10.5

During ordinary training, each step uses a different network architecture. At prediction time, standard dropout is turned off and the full network produces predictions. A Bayesian interpretation instead treats each dropped or retained unit as a Bernoulli choice, creating a distribution of networks. section 10.5 section 10.6

Monte Carlo dropout

To estimate uncertainty, keep dropout active during prediction and run the same input through the network repeatedly. Each forward pass removes a different subset of units; the resulting predictions form an approximate predictive distribution. With PyTorch, 2,000 forward passes can produce predictive samples whose mean and standard deviation summarize the result. section 10.6 section 10.7

Figure 10.10 Illustration of dropout. Left: a standard neural network. Right: After dropout is applied, specific nodes are randomly selected to be dropped for a given training step, producing zero output.

Compare a standard neural network with one training step after randomly dropping units and setting their outputs to zero. section 10.5

import torch
import torch.nn as nn
 
class DropoutBNN(nn.Module):
    def __init__(self, n_hidden, p_dropout):
        super().__init__()
        self.fc1 = nn.Linear(1, n_hidden)
        self.dropout = nn.Dropout(p_dropout)  #A
        self.fc2 = nn.Linear(n_hidden, 1)
    
    def forward(self, x):
        hidden = torch.tanh(self.fc1(x))
        hidden = self.dropout(hidden)         #B
        mu = self.fc2(hidden)
        return mu

Inspect a minimal PyTorch network that inserts a dropout layer between hidden and output layers. section 10.7

How Grokking Bayes covers it

How deep it goes
Explains dropped units, Bernoulli architecture samples, stochastic prediction, and PyTorch implementation without deriving the full variational objective.
Examples
Uses dropped-node diagrams, a PyTorch dropout layer, and 2,000 repeated forward passes to generate predictive samples.
What it assumes
Assumes basic neural-network training and prediction, while introducing regularization and Bayesian uncertainty through concrete dropout behavior.
What you can do after
Operate/configure it by adding PyTorch dropout and repeated prediction passes, then compute predictive means and standard deviations.
How it connects
Connects regularization with Bayesian inference, variational inference, predictive distributions, and neural-network training.

Also explained in

Compare how 16 books explain dropout →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Bayes. The section links point to the parts of the book it draws on.