Vision Transformer

As explained in How Large Language Models Work

A vision transformer is a transformer-based architecture that processes image data by representing it as a sequence of visual elements.

A vision transformer (ViT) applies transformer processing to images by turning visual data into a sequence of numeric vectors. Instead of treating an image as a sentence, it breaks the image into small square patches, processes those patches together, and uses their relationships to build a coherent interpretation or output. Transformers are useful here because they can connect information across the whole image rather than only nearby pixels. section 6.3 section 6.3.1

A typical image-to-image path works like this:

• Split the image into square tiles called patches. section 6.3.1
• Convert each patch’s pixel values into a vector of numbers. section 6.3.1
• Use a small preprocessing network, such as a CNN or linear layer, before passing the vectors to the transformer. section 6.3.1
• Convert transformer outputs back into patch-shaped vectors and combine them into an image. section 6.3.1

This replaces text tokenization and unembedding with image-specific patch extraction and patch combination. No image vocabulary or sampling process is required. The same general pattern also supports multimodal systems: images can become text descriptions, text can become images, or different input and output modalities can be combined. section 6.3.1 section 6.3.2

From image understanding to image generation

An image-captioning model accepts an image and produces text such as “This picture shows a stop sign surrounded by murky water.” An image-generation model reverses the direction: a prompt such as “Draw me a picture of a stop sign in a flood zone” produces an image. These systems remain constrained by their training examples. A request such as “Draw anything but an adorable cat” may still produce a cat because the model learned strong associations between the word “cat” and cat imagery, not the abstract instruction to exclude it. section 6.3.2 section 6.3.3

Figure 6.12 On the left, this simplified diagram shows how text input is tokenized and embedded before going to the transformer. An unembedding layer then converts the transformer output into the desired text representation. The input and output will be images when performing a computer vision task. The transformer stays the same, but then we modify the method for breaking the image into a sequence of vectors to perform patch extraction instead of tokenization. The LLM produces image output using a patch combiner, analogous to the unembedding layer for text LLMs.

Compare the text-model pipeline with the vision pipeline: tokenization becomes patch extraction, and unembedding becomes patch combination. section 6.3.1

How How Large Language Models Work covers it

How deep it goes

Traces images through patches, preprocessing, transformer vectors, and reconstruction, but does not derive transformer internals mathematically.

Examples

Uses patch diagrams, image captions, text-to-image prompts, and a cat-generation failure caused by contradictory instructions.

What it assumes

Assumes a basic transformer pipeline and introduces image patches as the replacement for text tokens.

What you can do after

Evaluate image and multimodal model behavior by tracing representations and checking requests against likely training patterns.

How it connects

Connects transformers with tokenization, convolutional neural networks, multimodal models, image captioning, and image generation.

liveBookmanning.com

Read How Large Language Models Work on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Build a Text-to-Image Generator (from Scratch)A Transformer model that processes an image as a sequence of patch representations for computer-vision tasks.
• 

  liveBookmanning.com
  Generative AI in ActionA vision transformer is a Transformer-based neural model that processes images by converting image regions into a sequence of representations and applying attention.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA transformer-based neural network adapted for computer vision tasks, especially image classification, by processing image patches with attention.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthA vision transformer is a transformer-based image model that processes an image as a sequence of embedded patches.
• 

  liveBookmanning.com
  AI-Powered SearchA vision transformer is a transformer-based neural model that represents and processes visual data such as images.
• 

  liveBookmanning.com
  Inside AIA vision transformer is a transformer-based model that processes image representations for computer-vision tasks.

Compare how 7 books explain Vision Transformer →

Related concepts

• transformer architecturekind of · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
• scaled-dot product attentionuses · Scaled dot-product attention computes query-key dot products, divides them by the square root of the key dimension, applies softmax, and uses the resulting weights to combine value vectors.
• patchuses · A patch is a localized segment or subregion of data treated as a unit for representation or processing.
• convolutional neural networkalternative to · A convolutional neural network is a neural-network architecture that applies learned convolutional operations to extract patterns from structured data.
• attentionuses · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.

This explainer was generated by Manning's knowledge graph from the text of How Large Language Models Work. The section links point to the parts of the book it draws on.