Patch

As explained in How Large Language Models Work

A localized rectangular portion of an image converted into a numerical representation for processing by a model.

A patch is a small, fixed-size piece of an image represented as numbers. Dividing an image into patches gives a transformer a sequence of inputs, much as tokenization gives a language model a sequence of text units. section 6.3.1 section 6.4

A typical patch-processing pipeline works like this:

• Divide the image into fixed-size tiles, commonly 16 × 16 pixels. section 6.3.1
• Convert each pixel’s red, green, and blue values into one long numerical vector; a 16 × 16 RGB patch contains 768 values. section 6.3.1
• Pass the vectors through a small neural network before sending the resulting sequence to the transformer. section 6.3.1

Figure 6.12 On the left, this simplified diagram shows how text input is tokenized and embedded before going to the transformer. An unembedding layer then converts the transformer output into the desired text representation. The input and output will be images when performing a computer vision task. The transformer stays the same, but then we modify the method for breaking the image into a sequence of vectors to perform patch extraction instead of tokenization. The LLM produces image output using a patch combiner, analogous to the unembedding layer for text LLMs.

Trace how patch extraction replaces tokenization while the transformer remains central, and how a patch combiner reconstructs image output. section 6.3.1

How How Large Language Models Work covers it

How deep it goes

Explains extraction, RGB vectorization, preprocessing, and reconstruction, but omits detailed neural-network mathematics.

Examples

Uses 16 × 16 RGB tiles, each converted into a 768-value vector.

What it assumes

Assumes familiarity with pixels and RGB values while explaining their conversion into vectors.

What you can do after

Evaluate how an image becomes transformer input and how model outputs can return to patch-shaped image data.

How it connects

Connects image patches with tokenization, embeddings, transformers, vectors, and multimodal models.

liveBookmanning.com

Read How Large Language Models Work on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Build a Text-to-Image Generator (from Scratch)An image patch is a spatially localized portion of an image represented separately in a grid or latent representation.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionAn image patch is a localized rectangular portion of an image processed as an individual visual input unit.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA local subregion of an image that is extracted and treated as one sequence element in ViT processing.
• 

  liveBookmanning.com
  Time Series Forecasting Using Foundation ModelsA patch is a segment of consecutive time-series observations grouped as one model input unit.

Compare how 5 books explain patch →

Related concepts

• Vision Transformerused by · A Vision Transformer is a Transformer-based neural-network architecture that processes images as sequences of tokenized regions using attention.
• PyTorch Unfoldused by · PyTorch Unfold is a tensor operation that extracts sliding local blocks from an input tensor.
• embeddingproduces · A numerical vector representation of an item or input that encodes features or relationships useful for computational tasks.
• encoderproduced by · An encoder is a component that transforms input data into an encoded representation for subsequent processing, reconstruction, comparison, or transmission.
• VQGAN decoderproduced by · A VQGAN decoder is the image-reconstruction component that maps discrete image tokens to an output image.

This explainer was generated by Manning's knowledge graph from the text of How Large Language Models Work. The section links point to the parts of the book it draws on.