liveBook · Manning

Codebook

As explained in Build a Text-to-Image Generator (from Scratch)

A codebook is a learned finite collection of discrete vectors used as representatives for continuous latent vectors.

A codebook is a learned, finite collection of vectors that represent continuous latent vectors. In a VQ-VAE or VQGAN, an encoder compresses an image into a grid of continuous feature vectors. A quantizer replaces each one with its nearest codebook vector. The position of that vector in the codebook is an integer code; the decoder uses the selected vectors to reconstruct the image. section 11.3.2 section 11.5.2

A standard embedding maps an input to a continuous vector. A codebook instead supplies a fixed-size set of learned prototypes from which the quantizer must choose. This is also the distinction from a conventional VAE’s continuous, probabilistic latent representation: a VQ-VAE represents encoded inputs with selected codebook entries. section 11.3.1 section 11.3.2

Trace an image through the codebook:

  • Encode the image into a lower-resolution grid of feature vectors. section 11.5.2
  • For each grid position, choose the nearest of the learned codebook vectors and record its index. The ordered indices form an image-token sequence. section 11.5.2
  • Look up the vectors by index, reshape them into a quantized latent grid, and pass that grid through the decoder to reconstruct the image. section 11.5.2

A 576-token image

A 384 × 384 image becomes a 24 × 24 latent grid: 576 patches, each with a 256-value feature vector. With 1,024 vectors in the codebook, quantization assigns one index to each patch. An example sequence begins 873, 862: those are the codebook entries nearest the first two latent vectors. In the pretrained VQGAN, `model.quantize(z)` returns quantized vectors and indices; `model.quantize.embedding(int_sequence)` retrieves vectors from the indices for decoding. section 11.5.2

Figure 11.8 Converting an image into a sequence of integers. The VQGAN model first encodes an image into a continuous latent representation, z, in the latent space, which consists of 576 feature vectors. Each of these vectors is matched with one of the 1,024 discrete feature vectors in the codebook. Therefore, the image can be represented by a sequence of integers, with each integer representing the index of the feature vector in the codebook.
A diagram of a diagram of a code

AI-generated content may be incorrect.

Follow the 576 continuous feature vectors as each selects one of 1,024 codebook entries, producing an integer sequence. section 11.5.2

The indices make images available to a transformer as sequences of discrete tokens rather than raw pixels. A text-to-image model can predict those tokens one at a time, then use a VQGAN decoder and its learned codebook to turn them into an image. Quantization has a cost: because a chosen prototype need not equal the original latent vector, reconstructed images can lose information. VQGAN training combines reconstruction, quantization, and adversarial losses to support useful codes and perceptually convincing images. section 12.1.2 section 11.5.1 section 11.4.2

How Build a Text-to-Image Generator (from Scratch) covers it

How deep it goes
Works through nearest-vector assignment and reverse lookup with a pretrained VQGAN; does not implement codebook training from scratch.
Examples
Traces a 384 × 384 image through 576 latent patches, 1,024 codebook entries, an index sequence, and reconstruction.
What it assumes
Builds the needed GAN and VQ-VAE context before using a pretrained model; code examples assume familiarity with PyTorch.
What you can do after
Apply it via a pretrained VQGAN to inspect image-token indices and reconstruct an image from codebook lookups.
How it connects
Connects vector quantization, latent representations, VQ-VAEs, VQGANs, and transformer image tokens.

Also explained in

Compare how 3 books explain codebook →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a Text-to-Image Generator (from Scratch). The section links point to the parts of the book it draws on.