Dropout regularizes a network by randomly replacing feature values with zero during training. Because any feature or neuron may disappear, the model cannot depend too heavily on one signal. section 7.4.1
Apply dropout at different strengths depending on where it enters the network:
- Use it on inputs, typically dropping 5–20% of values. section 7.4.1
- Use the default 50% probability more readily in hidden layers. section 7.4.1
- Insert dropout modules throughout an encoder and decoder, then train the denoising autoencoder normally. section 7.4.1
dnauto_encoder_dropout = nn.Sequential(
nn.Flatten(),
nn.Dropout(p=0.2), #1
getLayer(D, D*2),
nn.Dropout(), #2
getLayer(D*2, D*2),
nn.Dropout(),
getLayer(D*2, D*2), nn.Dropout(), nn.Linear(D*2, D*2)
)
dnauto_decoder_dropout = nn.Sequential(
getLayer(D*2, D*2),
nn.Dropout(),
getLayer(D*2, D*2),
nn.Dropout(),
getLayer(D*2, D*2),
nn.Dropout(),
nn.Linear(D*2, D),
View(-1, 1, 28, 28)
)
dnauto_encode_decode_dropout = nn.Sequential(
dnauto_encoder_dropout,
dnauto_decoder_dropout
)
train_network(dnauto_encode_decode_dropout, mse_loss, #3
train_loader, test_loader=test_loader, epochs=10, device=device)
#1 For the input, we usually drop only 5 to 20% of the values.
#2 By default, dropout uses 50% probability to zero out values.
#3 Trains as usual
Shows input dropout with p=0.2, default dropout in hidden layers, and the surrounding autoencoder architecture. section 7.4.1
