Dropout regularizes a neural network by randomly removing units during training. A removed unit outputs zero for that step, weakening co-adaptation and reducing overfitting so the network can generalize better. section 10.5
During ordinary training, each step uses a different network architecture. At prediction time, standard dropout is turned off and the full network produces predictions. A Bayesian interpretation instead treats each dropped or retained unit as a Bernoulli choice, creating a distribution of networks. section 10.5 section 10.6
Monte Carlo dropout
To estimate uncertainty, keep dropout active during prediction and run the same input through the network repeatedly. Each forward pass removes a different subset of units; the resulting predictions form an approximate predictive distribution. With PyTorch, 2,000 forward passes can produce predictive samples whose mean and standard deviation summarize the result. section 10.6 section 10.7
Figure 10.10 Illustration of dropout. Left: a standard neural network. Right: After dropout is applied, specific nodes are randomly selected to be dropped for a given training step, producing zero output.
Compare a standard neural network with one training step after randomly dropping units and setting their outputs to zero. section 10.5
import torch
import torch.nn as nn
class DropoutBNN(nn.Module):
def __init__(self, n_hidden, p_dropout):
super().__init__()
self.fc1 = nn.Linear(1, n_hidden)
self.dropout = nn.Dropout(p_dropout) #A
self.fc2 = nn.Linear(n_hidden, 1)
def forward(self, x):
hidden = torch.tanh(self.fc1(x))
hidden = self.dropout(hidden) #B
mu = self.fc2(hidden)
return mu
Inspect a minimal PyTorch network that inserts a dropout layer between hidden and output layers. section 10.7















