A discrete code is a latent representation selected from a finite learned codebook. Unlike a VAE's continuous, probabilistic latent vector, it uses the nearest codebook vector and can be stored as an integer index, improving interpretability and reconstruction sharpness. section 11.3.1 section 11.3.2
A VQ-VAE produces discrete codes through a sequential pipeline:
- The encoder compresses an input into a grid of continuous latent vectors. section 11.3.2
- Vector quantization replaces each vector with its nearest learned codebook embedding. section 11.3.1 section 11.3.2
- The selected embeddings form the discrete latent code, while their indices form a sequence of integers. section 11.3.2
- The decoder reconstructs the input from the quantized embeddings, using reconstruction and quantization losses during training. section 11.3.2
Encoding a fish image
A pretrained VQGAN converts a fish image into 576 integer tokens and decodes those tokens into a reconstruction that closely resembles the original. This demonstrates how an image's compressed visual information can be represented as a discrete sequence. section 11.1
Figure 11.5 The architecture of a VQ-VAE and its three components—encoder, codebook, and decoder—in the training process
Trace how the encoder, codebook, and decoder transform an image into quantized latent representations and back again. section 11.3.2
