Reconstruction loss measures how much an input changes after it is encoded into a latent vector and decoded back into input space. It supplies the training signal for preserving useful information under a constrained representation: a smaller latent space cannot retain every detail, so the model must favor what it can reconstruct well. section 14.0 section 14.5
Figure 14.5 Schematic representation of an autoencoder. The encoder transforms input into a latent vector. The decoder transforms the latent vector into reconstructed input. We minimize the reconstruction loss—the distance between the reconstructed input and the original input.
An autoencoder makes the comparison explicit: the encoder maps an input to a latent vector, the decoder produces a reconstruction, and their distance is minimized. section 14.5
For an autoencoder, an encoder–decoder pair is trained end to end so that reconstructed input is as close as possible to the original input. The target is implicit—the input itself—so training needs no human labels. Reconstruction loss therefore turns dimensionality reduction into representation learning: the latent-size budget is fixed, while the network learns which information to retain. section 14.5
Low reconstruction loss has useful but limited implications:
- In PCA, dropping directions with small principal values can produce low reconstruction loss because those directions carry little variance and may be noise. section 14.4
- A low loss alone does not select a unique latent geometry: smooth and zigzag manifolds can reconstruct training data equally well. section 14.6
- Regularization adds constraints beyond reconstruction, favoring compact, smooth latent spaces rather than merely accurate round trips. section 14.6
PyTorch autoencoder loss
After encoding X to z and decoding z to Xr, mean squared error directly computes reconstruction loss: `recon_loss = F.mse_loss(Xr, X)`. For MNIST, an ordinary autoencoder is trained with MSE between input and reconstructed images. A VAE example instead uses binary cross-entropy for its reconstruction term, which can converge better in practice. section 14.5 section 14.7.8
VAEs retain reconstruction loss but pair it with KL-divergence loss. The reconstruction term rewards decoding sampled latent vectors back near their inputs; the KL term pulls encoder-produced latent distributions toward a chosen prior. Together, these terms correspond to maximizing the ELBO and yield a more compact, regularized latent space than reconstruction alone. section 14.7.2 section 14.7.6 section 14.7.8
