Gaussian noise adds random perturbations to training data so an autoencoder learns to reconstruct clean inputs and robust representations. This is useful when real inputs contain camera damage, stains, lighting changes, or shadows, such as in OCR. section 7.4.1
A practical implementation inserts an additive-noise module before the encoder. The module adds noise during training and returns inputs unchanged during evaluation, while the network trains normally with reconstruction loss. section 7.4.1
Noise in an autoencoder
Place AdditiveGaussNoise() after Flatten and before the encoder layers, then compare reconstructions from clean and addNoise(input) data. The amount of Gaussian noise requires dataset-specific tuning, which makes dropout a more convenient alternative in many cases. section 7.4.1
class AdditiveGaussNoise(nn.Module): #1
def __init__(self):
super().__init__()
def forward(self, x):
if self.training: #2
return addNoise(x, device=device)
else: #3
return x
Inspect the training-only AdditiveGaussNoise module and its behavior in evaluation mode. section 7.4.1
dnauto_encoder_big = nn.Sequential(
nn.Flatten(),
AdditiveGaussNoise(), #1
getLayer(D, D*2),
getLayer(D*2, D*2),
getLayer(D*2, D*2),
nn.Linear(D*2, D*2),
)
dnauto_decoder_big = nn.Sequential(
getLayer(D*2, D*2),
getLayer(D*2, D*2),
getLayer(D*2, D*2),
nn.Linear(D*2, D),
View(-1, 1, 28, 28)
)
dnauto_encode_decode_big = nn.Sequential(
dnauto_encoder_big,
dnauto_decoder_big
)
train_network(dnauto_encode_decode_big, mse_loss, train_loader,
test_loader=test_loader, epochs=10, device=device) #2
#1 Only addition! We hope inserting noise here helps.
#2 Trains as usual
Follow the insertion of additive noise into a denoising autoencoder and its ordinary training call. section 7.4.1
