Visual encoder

As explained in Deep Learning with PyTorch, Second Edition

The part of a multimodal model that converts an image into a numeric representation for downstream decoding.

A visual encoder converts an image into a learned numerical representation that another model component can use. In image captioning, it turns scene content—such as a tabby cat, laser mouse, and paw—into an embedding for a text decoder. section 2.4

The encoder is usually the first half of a larger system. A decoder consumes its representation to generate text or a segmentation mask, while other inputs may be encoded alongside the image. section 2.4 section 15.4

Its role changes with the downstream task:

• Captioning: produce descriptive image embeddings that a text decoder transforms into a coherent sentence. section 2.4
• Segmentation: extract image features for a mask decoder, which combines them with encoded points, boxes, or masks. section 15.4
• SAM uses a Vision Transformer that divides an image into patches and applies self-attention to capture local and global relationships. section 15.4

From image to output

For captioning, an image encoder creates a numerical description of a scene, and a text decoder turns it into a sentence. For SAM, the encoder supplies image features while prompt information guides the mask decoder toward regions of interest. section 2.4 section 15.4

Figure 2.9 Conceptual diagram of BLIP caption generation

Use the conceptual diagram to follow the image representation from the encoder into the text decoder and generated caption. section 2.4

How Deep Learning with PyTorch, Second Edition covers it

How deep it goes

Explains embeddings, downstream decoding, and ViT self-attention, while deferring detailed encoder and decoder mechanics.

Examples

Uses BLIP-style captioning and SAM segmentation, including cat-scene embeddings and prompt-guided masks for a dog.

What you can do after

Apply pretrained image encoders by passing their representations to text or mask decoders for captions or segmentation.

How it connects

Connects image embeddings with text decoders, prompt encoders, mask decoders, segmentation, Vision Transformers, and self-attention.

liveBookmanning.com

Read Deep Learning with PyTorch, Second Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Deep Learning with R, Third EditionAn image encoder is a model component that transforms an input image into a lower-dimensional image embedding.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA model component that converts images into vector representations suitable for a larger multimodal model.
• 

  liveBookmanning.com
  Build a Text-to-Image Generator (from Scratch)An image encoder is a neural-network component that maps an image to a numerical representation.

Compare how 4 books explain visual encoder →

Related concepts

• CLIPpart of · CLIP is a multimodal model and pretraining method that aligns image and text representations in a shared embedding space.
• Gemmapart of · Gemma is a family of open-weight generative AI models developed by Google.
• SAMpart of · SAM, or Segment Anything Model, is a pretrained promptable computer-vision model that generates segmentation masks for objects in images.
• ONNXproduces · ONNX is an open standard for representing machine-learning models in a portable format that supports exchange and execution across frameworks and runtimes.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with PyTorch, Second Edition. The section links point to the parts of the book it draws on.