VQGAN

As explained in Build a Text-to-Image Generator (from Scratch)

A vector-quantized generative adversarial network that combines discrete latent representations with adversarial and perceptual image modeling.

VQGAN (vector-quantized generative adversarial network) turns an image into a sequence of integer codebook indices and reconstructs an image from that sequence. Its generator is a vector-quantized variational autoencoder (VQ-VAE); a discriminator helps its reconstructions look convincing. The discrete codes give transformers image tokens to predict instead of raw pixels. section 11.0 section 11.4.2 section 12.1.2

Trace an image through the model:

• Encode: the encoder compresses the image into a spatial grid of continuous feature vectors. section 11.5.2
• Quantize: replace each vector with its nearest learned codebook vector. Record that vector’s integer index; the grid becomes a token sequence. section 11.5.2
• Decode: look up the vectors by their indices, restore the quantized grid, and pass it through the decoder to reconstruct the image. section 11.5.2
• Train for appearance as well as reconstruction: the loss combines reconstruction, adversarial, and quantization terms. Perceptual reconstruction loss can use LPIPS to compare how images look, rather than only their pixel values. section 11.4.2

Figure 11.6 VQGAN architecture and training process in which VQGAN combines a VQ-VAE generator with a discriminator in a GAN framework

The architecture diagram separates the VQ-VAE generator’s encoder, codebook, and decoder from the discriminator used during adversarial training. section 11.4.2

Follow a cat image into tokens

With the illustrated pretrained model, a 384 × 384 image becomes a latent tensor of shape [1, 256, 24, 24]. Four stages of downsampling leave 576 spatial positions. Each 256-value vector is matched to one of 1,024 codebook entries, yielding 576 integers. Looking up those entries and decoding them reconstructs the image; comparing the result with the original makes the compression visible. section 11.5.2 section 11.5.1

z_q, _ , indices= model.quantize(z)    #1
int_sequence = indices[2]    #2
print(int_sequence)

The quantization call exposes both the selected latent vectors and the integer indices used as image tokens. section 11.5.2

z_q2 = model.quantize.embedding(int_sequence)
z_q2 = z_q2.permute(1,0).view(z_q.shape)    #1
z_post2 = model.post_quant_conv(z_q2)
z_recon2 = model.decoder(z_post2)    #2

This reverse path looks up vectors from the saved indices, reshapes them into a latent grid, and calls the decoder. section 11.5.2

In a transformer-based text-to-image pipeline, VQGAN supplies the image-token targets during training. At generation time, a text-conditioned transformer predicts tokens and the VQGAN decoder turns them into an image. This division makes sequential image generation practical without asking the transformer to predict every pixel directly. section 1.2.2 section 12.1.2

How Build a Text-to-Image Generator (from Scratch) covers it

How deep it goes

Traces encoding, codebook quantization, decoding, and training losses, then inspects a pretrained implementation rather than training VQGAN from scratch.

Examples

Reconstructs cat and fish images and converts a 384 × 384 cat image into 576 codebook indices.

What it assumes

Introduces GAN and VQ-VAE fundamentals before VQGAN, while the hands-on reconstruction uses Python and PyTorch.

What you can do after

Apply it via a pretrained PyTorch VQGAN to inspect image tokens and reconstruct images from their indices.

How it connects

Connects VQ-VAE, GAN discriminators, learned codebooks, perceptual loss, and autoregressive transformers.

liveBookmanning.com

Read Build a Text-to-Image Generator (from Scratch) on liveBook Get the book at manning.com

Related concepts

• generative adversarial networkkind of · A generative adversarial network is a generative model with a generator and discriminator trained in competition, enabling the generator to produce samples resembling a target dataset or distribution.
• decodehas part · Decoding is the process of transforming an encoded or internal representation into an interpretable or target-form output.
• encoderhas part · An encoder is a component that transforms input data into an encoded representation for subsequent processing, reconstruction, comparison, or transmission.
• reconstructed imageproduces · A reconstructed image is an output produced by decoding a compressed or latent representation to approximate an original input image.
• vector quantized variational autoencoderuses · A vector quantized variational autoencoder is a variational autoencoder that represents inputs with discrete learned codebook vectors.

This explainer was generated by Manning's knowledge graph from the text of Build a Text-to-Image Generator (from Scratch). The section links point to the parts of the book it draws on.