Vector quantized variational autoencoder

As explained in Build a Text-to-Image Generator (from Scratch)

A vector quantized variational autoencoder is a variational autoencoder that represents inputs with discrete learned codebook vectors.

A vector-quantized variational autoencoder (VQ-VAE) represents inputs with discrete latent codes selected from a learned codebook. Unlike a traditional variational autoencoder, it replaces continuous probabilistic latent distributions with finite, interpretable embeddings, reducing blurry reconstructions and inefficient latent-space use. section 11.3 section 11.3.1

Trace an input through the VQ-VAE:

• Encode the input into a grid of continuous latent vectors rather than directly producing discrete values. section 11.3.2
• Quantize each latent vector by replacing it with the nearest learned vector in the finite codebook. section 11.3.2
• Represent each selected codebook vector by its integer index, producing a sequence or grid of discrete latent codes. section 11.3.2
• Decode the quantized vectors to reconstruct the original input, optimizing reconstruction and quantization losses during training. section 11.3.2

The encoder, codebook, and decoder form the core architecture. The codebook contains learned prototype vectors, so quantization removes the continuous sampling uncertainty associated with ordinary VAEs. The resulting discrete representation supports sharper reconstruction and efficient downstream generation, including image synthesis and text-to-image systems. section 11.3.1 section 11.3.2

From image to tokens

An image such as a cat is compressed into a grid of latent feature vectors. Quantization replaces every vector with its nearest codebook entry, turning the image into integer indices that the decoder converts back into an image. section 11.3.2

Figure 11.5 The architecture of a VQ-VAE and its three components—encoder, codebook, and decoder—in the training process

Use the architecture figure to follow the encoder-to-codebook-to-decoder path and the associated training losses. section 11.3.2

How Build a Text-to-Image Generator (from Scratch) covers it

How deep it goes

Explains encoder, codebook, decoder, quantization, and losses, but defers full from-scratch implementation and training.

Examples

Uses a cat image, latent feature grids, codebook indices, and reconstructed images to illustrate encoding and decoding.

What it assumes

Assumes familiarity with autoencoders and VAEs, while contrasting their continuous latent distributions with discrete codebooks.

What you can do after

Evaluate/understand how discrete image representations support sharper reconstruction and token-based generative models.

How it connects

Connects variational autoencoders, vector quantization, codebooks, image synthesis, VQGAN, and transformer-based text-to-image generation.

liveBookmanning.com

Read Build a Text-to-Image Generator (from Scratch) on liveBook Get the book at manning.com

Related concepts

• variational autoencoderkind of · A variational autoencoder is a generative neural network that learns a probabilistic latent representation of data and decodes samples from it to reconstruct or generate data.
• image tokenizationused by · Image tokenization is the process of converting image content into a sequence of discrete visual tokens.
• codebookuses · A codebook is a collection that maps codes to their meanings or to representative values used to encode data.
• reconstruction lossuses · Reconstruction loss is a measure of the discrepancy between an input or observation and the output a model reconstructs from it or its representation.
• VQGANused by · VQGAN is a generative model that combines vector-quantized discrete latent representations with adversarial and perceptual objectives to synthesize images.
• vector-wise quantizationuses · Vector-wise quantization maps each input vector to a representative vector selected from a finite codebook.

This explainer was generated by Manning's knowledge graph from the text of Build a Text-to-Image Generator (from Scratch). The section links point to the parts of the book it draws on.