liveBook · Manning

Monte Carlo dropout

As explained in Grokking Bayes

MC dropout is a neural-network inference technique that retains dropout during prediction and uses repeated stochastic forward passes to approximate a predictive distribution.

A neural network can give different answers for the same uncertain case. Monte Carlo dropout estimates that uncertainty by keeping dropout active during prediction and repeating the forward pass. Each pass samples a slightly different network, producing a collection of predictions. Their average gives a mean estimate, while their spread estimates predictive uncertainty. section 10.6 section 10.9

Apply Monte Carlo dropout in a few practical steps:

  • Train the dropout network with ordinary gradient-based optimization. section 10.7
  • Leave dropout enabled when making predictions; in PyTorch, avoid calling model.eval(). section 10.7
  • Run many forward passes for each test input and collect the resulting predictions. section 10.7
  • Compute the sample mean and standard deviation to summarize the prediction and its uncertainty. section 10.6 section 10.7

A PyTorch prediction loop

A model can define an ordinary dropout layer, then sample predictions repeatedly: `T = 2000; predictions = torch.stack([model(x_test) for _ in range(T)])`. The resulting tensor contains approximate predictive samples from which the mean and standard deviation can be calculated. section 10.7

Figure 10.12 At test time, for a given input, we perform multiple forward passes with dropout enabled to obtain a distribution of predictions, which serves as our predictive distribution for that input.

Visualizes how multiple dropout-enabled forward passes turn one input into a distribution of predictions. section 10.6

How Grokking Bayes covers it

How deep it goes
Explains the Bayesian interpretation, prediction procedure, PyTorch implementation, and trade-offs, but not deeper posterior derivations.
Examples
Uses a PyTorch dropout network, 2,000 predictive forward passes, and visualized prediction distributions.
What it assumes
Assumes familiarity with training and predicting with neural networks, while introducing the Bayesian interpretation of dropout.
What you can do after
Apply it via PyTorch to generate predictive samples and summarize uncertainty with means and standard deviations.
How it connects
Connects dropout with Bayesian neural networks, MCMC, variational inference, and regularization.

Also explained in

Compare how 2 books explain Monte Carlo dropout →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Bayes. The section links point to the parts of the book it draws on.