Autoencoder

As explained in Math and Architectures of Deep Learning

A neural network model trained to compress inputs into a latent representation and reconstruct the original inputs from that representation.

An autoencoder is a neural network that learns a compact latent representation of an input and then reconstructs the input from that representation. Its encoder maps input to a lower-dimensional latent vector; its decoder maps that vector back to input space. Because the latent space has a limited size budget, reconstruction is necessarily lossy: training retains the information most useful for matching the original. section 14.5 section 14.0

Training is self-supervised in form: the desired output is the input itself, so labels are unnecessary. The encoder-decoder pair is trained end to end by minimizing reconstruction loss, the difference between the original input X and reconstructed output Xr. Mean squared error is one option; binary cross-entropy can compare normalized images or vectors with values from 0 to 1. section 14.5 section 9.1.4

Figure 14.5 Schematic representation of an autoencoder. The encoder transforms input into a latent vector. The decoder transforms the latent vector into reconstructed input. We minimize the reconstruction loss—the distance between the reconstructed input and the original input.

The diagram makes the bottleneck explicit: input passes through a latent vector before reconstruction loss compares the decoder output with the original. section 14.5

For images, a convolutional encoder progressively reduces height and width through convolution, ReLU, and max-pooling layers, then flattens features into a latent vector. A decoder reverses the spatial direction with transposed convolutions, increasing resolution until it produces an image-shaped output. Learned transposed-convolution upsampling can minimize final reconstruction error rather than relying on a predefined interpolation rule. section 14.5 section 10.5.3

A compact PyTorch training path

Encode an image batch X into convolutional features, project them to z, decode z into Xr, and compute recon_loss = mse_loss(Xr, X). One illustrated architecture reduces a 1 × 32 × 32 image through convolution and pooling layers to a latent vector of size 10, then uses transposed convolutions and a sigmoid output to reconstruct the image. section 14.5

Autoencoders generalize the dimensionality-reduction idea behind PCA. PCA is linear and projects onto best-fit hyperplanes, whereas an autoencoder with nonlinear layers can fit complex nonlinear hypersurfaces, limited by the encoder and decoder's expressive power. With only single linear encoder and decoder layers, an autoencoder behaves like a hyperplane projection. section 14.5.1 section 14.1

A plain autoencoder optimizes reconstruction loss without restricting the shape of its latent space. A variational autoencoder adds latent-space regularization through KL divergence to a prior distribution, producing a more compact latent space while retaining a reconstruction objective. section 14.7.8 section 14.8

How Math and Architectures of Deep Learning covers it

How deep it goes

Develops encoder-decoder training, reconstruction losses, convolutional implementations, and nonlinear comparison with PCA; defers autoencoder training details during transpose-convolution discussion.

Examples

Uses 32 × 32 image encoders and decoders, normalized image reconstruction, transpose-convolution upsampling, and MNIST reconstruction comparisons.

What you can do after

Build an image autoencoder in PyTorch with convolutional encoding, transposed-convolution decoding, and reconstruction loss.

How it connects

Connects latent vectors, dimensionality reduction, reconstruction loss, PCA, transposed convolution, and variational autoencoders.

liveBookmanning.com

Read Math and Architectures of Deep Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA neural network trained to reconstruct its input, typically by compressing it into a lower-dimensional latent representation and decoding it back.
• 

  liveBookmanning.com
  Outlier Detection in PythonAn autoencoder is a neural network trained to encode an input into a representation and reconstruct the original input.
• 

  liveBookmanning.com
  Data Without LabelsAn autoencoder is a neural-network model that learns to encode data into a lower-dimensional representation and reconstruct the original input from it.
• 

  liveBookmanning.com
  Evolutionary Deep Learning: Genetic algorithms and neural networksAn autoencoder is a neural network that learns to encode inputs into a representation and decode that representation to reconstruct the inputs.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorchA neural network that learns to compress inputs into a latent representation and reconstruct them, used here for digit-image generation and reconstruction.
• 

  liveBookmanning.com
  Graph Neural Networks in ActionAn autoencoder is a neural network that encodes inputs into latent representations and decodes those representations to reconstruct the inputs.
• 

  liveBookmanning.com
  Delivering Machine Learning Projects: From design to deploymentAn autoencoder is a neural network that learns to encode inputs into a lower-dimensional representation and reconstruct the original inputs.
• 

  liveBookmanning.com
  Financial AI in PracticeAn autoencoder is a neural network trained to reconstruct its input, typically by encoding it into a lower-dimensional representation and decoding it.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionAn autoencoder is a neural-network model that learns to encode inputs into representations and decode those representations back into approximations of the inputs.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA neural network trained to reconstruct its input, often by compressing and then decoding it, used here for denoising images.
• 

  liveBookmanning.com
  Introduction to Generative AI, Second EditionAn autoencoder is a neural network that learns to encode data into a compressed representation and reconstruct the original data from it.

Compare how 12 books explain autoencoder →

Related concepts

• encoderhas part · An encoder is a component that transforms input data into an encoded representation for subsequent processing, reconstruction, comparison, or transmission.
• decoderhas part · A decoder is a component that maps encoded data or representations to decoded values, reconstructed data, or generated output.
• variational autoencoderincludes · A variational autoencoder is a generative neural network that learns a probabilistic latent representation of data and decodes samples from it to reconstruct or generate data.
• deep learningkind of · Deep learning is a machine-learning approach that uses neural networks with multiple computational layers to learn representations from data.
• neural networkkind of · A neural network is a computational model of interconnected, parameterized units that learns transformations from inputs to outputs.
• Kerasuses · Keras is an open-source, high-level deep-learning API for building, training, evaluating, and deploying neural-network models.

This explainer was generated by Manning's knowledge graph from the text of Math and Architectures of Deep Learning. The section links point to the parts of the book it draws on.