Computer vision enables computational systems to analyze and interpret visual data such as images and videos. It supports tasks including object detection, object identification, image interpretation, and image generation. section 6.0
Transformers can work with images without changing their core layers. Instead of treating words as tokens, the system converts an image into a sequence of numeric vectors, processes that sequence, and converts the result back into image data. section 6.0 section 6.3
A transformer-based vision pipeline typically works as follows:
- Break the image into square tiles called patches. section 6.3.1
- Represent each pixel by its red, green, and blue values, then combine each patch into a long vector. section 6.3.1
- Use a small neural network to extract basic local features before the full transformer processes the vectors. section 6.3.1
- Use a patch combiner to reshape transformer outputs into patches when producing an image. section 6.3.1
A 16-by-16 patch
A 16 × 16 pixel patch with three RGB values per pixel becomes a vector containing 768 values. A small network can process this vector, while later layers combine processed vectors into an image representation or output. section 6.3.1
Figure 6.12 On the left, this simplified diagram shows how text input is tokenized and embedded before going to the transformer. An unembedding layer then converts the transformer output into the desired text representation. The input and output will be images when performing a computer vision task. The transformer stays the same, but then we modify the method for breaking the image into a sequence of vectors to perform patch extraction instead of tokenization. The LLM produces image output using a patch combiner, analogous to the unembedding layer for text LLMs.
Trace the analogy between text tokenization and image patch extraction, then compare text unembedding with image patch combination. section 6.3.1
Vision transformers use this arrangement to give the model global cohesion across an image. Other systems combine transformers with architectures such as convolutional neural networks, VQGAN, or U-Net transformer designs. section 6.3 section 6.3.1
