A digital image is organized numeric data: pixels store color values, commonly as red, green, and blue (RGB). For transformer processing, an image can be divided into small square patches, converted into vectors, and treated somewhat like a sequence of tokens. section 6.3.1
A patch-based representation works as follows:
- Divide the image into fixed-size patches, such as 16 × 16 pixel squares. section 6.3.1
- Convert every pixel’s red, green, and blue values into one long numeric vector. section 6.3.1
- Pass the patch vectors through a small preprocessing network before the transformer. section 6.3.1
A 16 × 16 patch
A 16 × 16 patch has 256 pixels. With three RGB values per pixel, its initial vector contains 768 values. section 6.3.1
Figure 6.12 On the left, this simplified diagram shows how text input is tokenized and embedded before going to the transformer. An unembedding layer then converts the transformer output into the desired text representation. The input and output will be images when performing a computer vision task. The transformer stays the same, but then we modify the method for breaking the image into a sequence of vectors to perform patch extraction instead of tokenization. The LLM produces image output using a patch combiner, analogous to the unembedding layer for text LLMs.
Shows how patch extraction replaces tokenization for image input while the transformer remains central, and how a patch combiner reconstructs image output. section 6.3.1
