Vector-wise quantization replaces a continuous latent vector with its nearest vector in a finite, learned codebook. The replacement is a quantized vector; its codebook index is an integer token. In an image model, applying this operation across a grid of latent vectors turns an image representation into a sequence of tokens. section 11.3.2 section 1.2.1
A VQ-VAE or VQGAN uses an encoder to compress an image into a grid of continuous feature vectors. Each vector selects its nearest codebook entry. The decoder reconstructs an image from the selected vectors, while the indices provide a compact discrete representation. Unlike a traditional VAE’s continuous, distribution-based latents, these latents are selected from a finite set. section 11.3.1 section 11.3.2
From a latent grid to image tokens
An encoder reduces a 384 × 384 image to a 24 × 24 grid of 256-dimensional vectors. Matching each of its 576 vectors against a codebook of 1,024 entries yields 576 indices. An index such as 873 means that the corresponding latent vector was matched to codebook entry 873—not that a pixel has value 873. section 11.5.2
z_q, _ , indices= model.quantize(z) #1 int_sequence = indices[2] #2 print(int_sequence)
Inspect the quantization call: it returns quantized vectors and the indices that form the integer sequence. section 11.5.2
To reverse the token representation, look up each index in the same codebook, reshape the retrieved vectors into the latent grid, and pass that grid to the decoder. This makes the indices useful to a transformer: it can predict image tokens, which the VQGAN decoder can turn into an image. Training the VQ-VAE also minimizes reconstruction and quantization losses. section 11.5.2 section 11.3.2
