A feature vector is a numerical vector whose components encode properties of an input or intermediate representation. In VQGAN, an encoder converts a 384 × 384 image into a 24 × 24 grid containing 576 feature vectors, each with 256 values. Each vector represents one spatial patch in the latent image. section 11.5.2
Feature vectors participate in a sequence of transformations:
- The encoder applies four strided convolutional layers, reducing spatial resolution by a factor of 16 and producing the 24 × 24 latent grid. section 11.5.2
- A codebook stores 1,024 learnable 256-dimensional vectors. Quantization replaces each continuous latent feature vector with its nearest codebook vector. section 11.5.2 section 11.3.2
- The selected codebook position becomes an integer index, turning the 576 feature vectors into a sequence of 576 integers. section 11.5.2
- Looking up those indices reconstructs the quantized latent representation, which the decoder uses to reconstruct the image. section 11.5.2
Inspecting and reconstructing latent vectors
For a cat image, encode the image and inspect a latent tensor with shape [1, 256, 24, 24]. Quantize its 576 vectors, obtain indices such as 873 and 862, then retrieve the corresponding codebook vectors and pass them through the decoder. Reconstructions from the quantized vectors and from the integer sequence can be compared directly. section 11.5.2 section 11.5
z=model.encoder(image1.to(device))
z=model.quant_conv(z)
print(f"the shape of the latent representation is {z.shape}")
Inspect the encoder output directly to connect tensor dimensions with the number and size of feature vectors. section 11.5.2
