A patch is a local image subregion treated as one unit of processing. Convolutions extract patches from feature maps and apply the same transformation at each location, helping models detect spatially repeated patterns such as edges and textures. section 8.1.1
Patches as image sequence elements
A Vision Transformer splits an image into a one-dimensional sequence of patches, flattens each patch into a vector, and processes the resulting sequence. This lets the Transformer model relationships between distant image regions. section 9.6
Use these distinctions when interpreting patches:
- Treat a convolution patch as a spatial window whose transformed responses remain arranged in a feature map. section 8.1.1
- Treat a ViT patch as a flattened sequence element rather than as a permanently two-dimensional feature-map location. section 9.6
- Treat an image patch in a multimodal model as a possible soft token after an image encoder converts it to a vector. section 16.4.2
Gemma image representation
Gemma’s image encoder cuts an image into 256 patches and encodes them as a 256 × 2560 sequence. The resulting vectors can be spliced into the text sequence as 256 image soft tokens. section 16.4.2
Figure 8.1 Images can be broken into local patterns such as edges, textures, and so on.
Use the image-pattern illustration to connect local patches with visual features such as edges and textures. section 8.1.1
