Quantization loss measures how far continuous encoder outputs remain from their selected discrete codebook vectors. It keeps latent representations aligned with the codebook used to produce discrete image codes. section 11.4.2 section 11.3.2
Within a VQ-VAE or VQGAN, quantization loss supports the following process:
- The encoder maps an image to a grid of continuous latent vectors. section 11.3.2
- Quantization replaces each latent vector with its nearest embedding from a learned codebook. section 11.3.2
- The loss encourages each encoder output z to stay close to its quantized counterpart zq, so the encoder consistently uses the codebook. section 11.4.2
- VQGAN combines quantization loss with reconstruction and adversarial losses; perceptual loss such as LPIPS can provide the reconstruction component. section 11.4.2 section 11.6
Image-to-code example
An image can be encoded into 576 continuous feature vectors arranged on a 24 × 24 grid. Each vector is matched against 1,024 codebook embeddings, producing discrete indices. Quantization loss trains the encoder to produce vectors that remain close to those selected embeddings while the decoder reconstructs the image. section 11.3.2
Figure 11.5 The architecture of a VQ-VAE and its three components—encoder, codebook, and decoder—in the training process
Trace the encoder, codebook, and decoder to see where quantization loss operates in the VQ-VAE training loop. section 11.3.2
