A sequence is data arranged so that each element’s position matters. An image can become a sequence of patches or tokens: swapping two patches changes the image they describe. A sequence of token values alone does not necessarily tell a model where those tokens belong, so position must also be represented. section 3.3.2
For image classification with a vision transformer, patch embeddings form 64 image tokens. A classification token is placed first, producing 65 tokens. The model learns a positional vector for each slot and adds it to the corresponding token embedding. With 48 values per token, both the token embeddings and positional embeddings have shape (1, 65, 48) for one image. section 3.3.2
Listing 3.3 Adding positional encoding to the image embedding
class Embeddings(nn.Module):
def __init__(self, config):
super().__init__()
self.config = config
self.patch_embeddings = PatchEmbeddings(config)
self.cls_token = nn.Parameter(torch.randn(1, 1, #1
config.hidden_size))
num_patches = (config.image_size // config.patch_size) ** 2
self.position_embeddings = \ #2
nn.Parameter(torch.randn(1,num_patches+1,config.hidden_size))
def forward(self, x):
x = self.patch_embeddings(x) #3
batch_size, _, _ = x.size()
cls_tokens = self.cls_token.expand(batch_size, -1, -1)
x = torch.cat((cls_tokens, x), dim=1) #4
x = x + self.position_embeddings #5
return x
The embedding implementation makes the order-bearing step visible: create patch tokens, prepend the classification token, then add learned positional embeddings. section 3.3.2
A different image sequence uses integers rather than patch embeddings. A VQGAN encoder compresses an image into a grid of continuous feature vectors. Quantization replaces each vector with its nearest entry in a learned codebook; the entry’s index becomes an integer in the sequence. Given that sequence and the codebook, the indices can be looked up, arranged back into a quantized latent grid, and passed through the decoder to reconstruct an image. This discrete form lets transformer-based text-to-image models treat images analogously to sequences of text tokens. section 1.2.1 section 11.5.2
Encode and rebuild a 576-element image sequence
A 24 × 24 latent grid contains 576 feature vectors. Match each vector to one of 1,024 codebook entries to obtain 576 indices. In the pretrained VQGAN workflow, `model.quantize(z)` supplies the indices; looking up their embeddings, reshaping them to the quantized latent grid, and running the decoder reconstructs the image. Compare the result with the original: discrete approximation can lose some information. section 11.5.2 section 11.5.1
z_q2 = model.quantize.embedding(int_sequence) z_q2 = z_q2.permute(1,0).view(z_q.shape) #1 z_post2 = model.post_quant_conv(z_q2) z_recon2 = model.decoder(z_post2) #2
The reverse-path listing shows how codebook indices become a latent grid and then a decoded image. section 11.5.2
