Vision Transformer

As explained in Generative AI in Action

A vision transformer is a Transformer-based neural model that processes images by converting image regions into a sequence of representations and applying attention.

A Vision Transformer (ViT) applies Transformer architecture to visual data. It suits flexible, sequence-based image tasks and image recognition, but requires significant computational resources. section 4.1 section 4.1.3

Instead of comparing every pixel with every other pixel, ViT divides an image into small patches, commonly 16 × 16 pixels. It flattens those patches, adds positional embeddings, and feeds the resulting sequence into a Transformer encoder. section 4.1.3

A ViT processes an image through three practical stages:

• Projects flattened image patches into vectors that form a linear input sequence. section 4.1.3
• Uses global self-attention to learn relationships among different image regions simultaneously. section 4.1.3
• Applies multi-head attention and normalization layers before assigning an output class such as Bird, Ball, or Car. section 4.1.3
• Scales attention cost quadratically with the number of input patches, making patch selection and compute capacity important considerations. section 4.1.3

Figure 4.6 Vision transformer (ViT) architecture [2]

Inspect the architecture diagram to connect patch projection, the Transformer encoder, attention layers, normalization, and output classes. section 4.1.3

Classifying an image

An image is divided into 16 × 16 regions. Each region becomes a vector with positional information, and the encoder learns relationships across the full sequence before selecting a class such as Bird, Ball, or Car. section 4.1.3

How Generative AI in Action covers it

How deep it goes

Explains patch tokenization, positional embeddings, global attention, encoder layers, and quadratic cost without mathematical derivation.

Examples

Uses 16 × 16 image patches and Bird, Ball, and Car class labels to illustrate visual classification.

What it assumes

Builds on Transformer architecture and attention mechanisms while defining their adaptation from sequential text or speech to images.

What you can do after

Evaluate/understand how ViTs convert images into patch sequences and trade global attention against computational cost.

How it connects

Links Transformer architecture with attention mechanisms, positional embeddings, image recognition, and Transformer encoders.

liveBookmanning.com

Read Generative AI in Action on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Build a Text-to-Image Generator (from Scratch)A Transformer model that processes an image as a sequence of patch representations for computer-vision tasks.
• 

  liveBookmanning.com
  How Large Language Models WorkA vision transformer is a transformer-based architecture that processes image data by representing it as a sequence of visual elements.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA transformer-based neural network adapted for computer vision tasks, especially image classification, by processing image patches with attention.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthA vision transformer is a transformer-based image model that processes an image as a sequence of embedded patches.
• 

  liveBookmanning.com
  AI-Powered SearchA vision transformer is a transformer-based neural model that represents and processes visual data such as images.
• 

  liveBookmanning.com
  Inside AIA vision transformer is a transformer-based model that processes image representations for computer-vision tasks.

Compare how 7 books explain Vision Transformer →

Related concepts

• transformer architecturekind of · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
• scaled-dot product attentionuses · Scaled dot-product attention computes query-key dot products, divides them by the square root of the key dimension, applies softmax, and uses the resulting weights to combine value vectors.
• patchuses · A patch is a localized segment or subregion of data treated as a unit for representation or processing.
• convolutional neural networkalternative to · A convolutional neural network is a neural-network architecture that applies learned convolutional operations to extract patterns from structured data.
• attentionuses · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.

This explainer was generated by Manning's knowledge graph from the text of Generative AI in Action. The section links point to the parts of the book it draws on.