Autoencoder

An autoencoder is a type of neural network designed to learn efficient codings of input data. It is primarily used for dimensionality reduction and feature learning. The architecture of an autoencoder consists of two main components: an encoder and a decoder. The encoder maps the input data to a lower-dimensional latent space, while the decoder reconstructs the input from this latent representation. The network is trained to minimize the reconstruction loss, which measures the difference between the original input and the reconstructed output. This process allows the autoencoder to perform nonlinear projections, making it more powerful than traditional methods like Principal Component Analysis (PCA) for dimensionality reduction.

Architecture

Encoder

The encoder is responsible for transforming the input data into a compact, lower-dimensional representation known as the latent vector. This transformation is achieved through a series of layers that progressively reduce the dimensionality of the input. An example of an encoding neural network for image inputs is shown in Listing 14.2. The encoder typically uses convolutional layers, activation functions like ReLU, and pooling layers to achieve this reduction.

Listing 14.2 PyTorch- Autoencoder encoder

from torch import nn
nz = 10
input_image_size = (1, 32, 32)            #1
conv_encoder = nn.Sequential(
            nn.Conv2d(in_channels, 32, kernel_size=3, stride=1, padding=1),
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2),  #2
            nn.Conv2d(32, 128, kernel_size=3, stride=1, padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2),  #3
            nn.Conv2d(128, 256, kernel_size=3, stride=1, padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2),  #4

            nn.Flatten()                  #5
        )
fc = nn.Linear(4096, nz)                  #6
#1 Input image size in (c, h, w) format
#2 Reduces to a (32, 16, 16)-sized tensor
#3 Reduces to a (128, 8, 8)-sized tensor
#4 Reduces to a (256, 4, 4)-sized tensor
#5 Flattens to a 4096-sized tensor
#6 Reduces the 4096-sized tensor to an nz-sized tensor

Decoder

The decoder takes the latent vector produced by the encoder and attempts to reconstruct the original input. This is done through a series of layers that progressively increase the dimensionality of the latent vector back to the original input size. The decoder often uses transposed convolutional layers, also known as deconvolutional layers, to achieve this upscaling. An example of a decoding neural network is shown in Listing 14.3.

Listing 14.3 PyTorch- Autoencoder decoder

from torch import nn
decoder = nn.Sequential(
            nn.ConvTranspose2d(self.nz, out_channels=256,
             kernel_size=4, stride=1,
             padding=0, bias=False),            #1

            nn.BatchNorm2d(256),
            nn.ReLU(True),
            nn.ConvTranspose2d(256, 128, kernel_size=2,
              stride=2, padding=0, bias=False), #2
            nn.BatchNorm2d(128),
            nn.ReLU(True),
            nn.ConvTranspose2d(128, 32, kernel_size=2,
              stride=2, padding=0, bias=False), #3
            nn.BatchNorm2d(32),
            nn.ReLU(True),
            nn.ConvTranspose2d(32, in_channels, kernel_size=2,
              stride=2, padding=0, bias=False), #4
            nn.Sigmoid()
        )
#1 Converts (nz, 1, 1) to a 256, 4, 4)-sized tensor
#2 Increases to a 128, 8, 8)-sized tensor
#3 Increases to a 32, 16, 16)-sized tensor
#4 Increases to a 1, 32, 32)-sized tensor

Training

Autoencoders are trained in an unsupervised manner, as the desired output is implicitly known: it is the input itself. The training process involves minimizing the reconstruction loss, which is often computed using mean squared error (MSE) between the input and the reconstructed output. The training process is illustrated in Listing 14.4.

Listing 14.4 PyTorch- Autoencoder training

from torch import nn
from torch.nn import functional as F

conv_out = conv_encoder(X)     #1

z =  fc(conv_out)              #2
Xr = decoder(z)                #3

recon_loss = F.mse_loss(Xr, X) #4
#1 Passes the input image through the convolutional encoder
#2 Reduces to nz dimensions
#3 Reconstructs the image using z via the decoder
#4 Computes the reconstruction loss

Schematic Representation

The schematic representation of an autoencoder is shown in Figure 14.5. The encoder transforms the input into a latent vector, and the decoder transforms the latent vector back into the reconstructed input. The goal is to minimize the reconstruction loss, which is the distance between the reconstructed input and the original input.

Figure 14.5 Schematic representation of an autoencoder. The encoder transforms input into a latent vector. The decoder transforms the latent vector into reconstructed input. We minimize the reconstruction loss—the distance between the reconstructed input and the original input. Figure 14.5 Schematic representation of an autoencoder. The encoder transforms input into a latent vector. The decoder transforms the latent vector into reconstructed input. We minimize the reconstruction loss—the distance between the reconstructed input and the original input.