An image patch is a spatially localized part of an image represented separately. Dividing an image into fixed-size patches lets a transformer process visual data as a sequence of tokens. The patches are ordered row by row, from top-left to bottom-right; unlike a word sequence, that order does not tell self-attention where each patch came from, so positional encodings supply spatial information. section 3.1.1
To turn patches into inputs for a vision transformer (ViT):
- Split the image into a grid. A 3 × 32 × 32 CIFAR-10 image becomes 64 patches of shape 3 × 4 × 4 when patch size is 4. section 3.3.1
- Project each patch into a vector. A PyTorch Conv2d layer with kernel size and stride both set to 4 produces a 48 × 8 × 8 tensor; flattening and transposing its grid gives 64 × 48 patch tokens. section 3.3.1
- For classification, prepend a class token and add positional encodings. The resulting sequence has 65 vectors of 48 values each; the class token gathers information for the image-level prediction. section 3.1.1
Listing 3.2 Converting image patches into a sequence
from torch import nn
class PatchEmbeddings(nn.Module):
def __init__(self, config):
super().__init__()
self.projection = nn.Conv2d( #1
config.num_channels,
config.hidden_size,
kernel_size=config.patch_size, #2
stride=config.patch_size) #3
def forward(self, x):
x = self.projection(x)
x = x.flatten(2).transpose(1, 2) #4
return x
The PatchEmbeddings implementation shows the projection and reshape: Conv2d extracts one vector per patch, then flatten and transpose put patches in the sequence dimension. section 3.3.1
Extract patches for an image-captioning encoder
For a 3 × 128 × 128 image, torch.nn.Unfold with an 8 × 8 kernel and stride 8 extracts a 16 × 16 grid. Transposing its output yields a batch tensor of shape (1, 256, 192): 256 patches, each containing 3 × 8 × 8 pixel values. A vision encoder can then project these vectors and pass their representations to a caption-generating decoder. section 4.3.1 section 4.5
A patch need not remain a vector of raw pixels. In VQGAN-based text-to-image generation, an encoder forms a grid of latent patch vectors, then replaces each with a learned codebook entry. The resulting integer image tokens can be predicted one at a time from a text prompt; a VQGAN decoder turns the token sequence back into an image. This differs from diffusion generation, which starts with noise and denoises iteratively rather than predicting a top-left-to-bottom-right patch sequence. section 1.2.1 section 1.7
