Visual encoder

As explained in Deep Learning with R, Third Edition

An image encoder is a model component that transforms an input image into a lower-dimensional image embedding.

A visual encoder transforms an input image into a smaller image embedding: a compact representation that other model components can process more easily. section 11.4.2

In an image-segmentation system, the visual encoder processes the image while a prompt encoder represents a point or box. A mask decoder combines both embeddings to predict masks. section 11.4.2

Visual encoders also adapt images for multimodal Transformers:

• A Gemma visual encoder divides an image into 256 patches and encodes each patch as a 2,560-dimensional vector. section 16.4.2
• The resulting image representation has shape (256, 2560), matching Gemma’s hidden Transformer dimension. section 16.4.2
• These vectors become 256 image soft tokens spliced into the text sequence after token embedding, allowing the Transformer to process image and text together. section 16.4.2

From image to mask or answer

For segmentation, an image embedding joins a point or box embedding and produces candidate object masks. For multimodal prompting, image soft tokens join a text prompt so Gemma can answer questions such as what a pictured snake is wearing. section 11.4.2 section 16.4.2

Figure 11.9 The SAM high-level architecture overview

Architecture overview showing how the image encoder, prompt encoder, and mask decoder cooperate. section 11.4.2

Figure 16.6 Handling image input by splicing together text tokens and soft image tokens

Diagram of image soft tokens being spliced into a text-token sequence. section 16.4.2

How Deep Learning with R, Third Edition covers it

How deep it goes

Explains embeddings, patch sequences, and component interfaces while deferring detailed encoder, prompt-encoder, and decoder techniques.

Examples

Uses SA-1B segmentation images, point and box prompts, Gemma image patches, and questions about a snake image.

What you can do after

Evaluate how images become embeddings, segmentation inputs, or soft tokens for multimodal Transformer processing.

How it connects

Connects image embeddings with prompt encoders, mask decoders, Transformers, soft tokens, image segmentation, and multimodal language models.

liveBookmanning.com

Read Deep Learning with R, Third Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA model component that converts images into vector representations suitable for a larger multimodal model.
• 

  liveBookmanning.com
  Build a Text-to-Image Generator (from Scratch)An image encoder is a neural-network component that maps an image to a numerical representation.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionThe part of a multimodal model that converts an image into a numeric representation for downstream decoding.

Compare how 4 books explain visual encoder →

Related concepts

• CLIPpart of · CLIP is a multimodal model and pretraining method that aligns image and text representations in a shared embedding space.
• Gemmapart of · Gemma is a family of open-weight generative AI models developed by Google.
• SAMpart of · SAM, or Segment Anything Model, is a pretrained promptable computer-vision model that generates segmentation masks for objects in images.
• ONNXproduces · ONNX is an open standard for representing machine-learning models in a portable format that supports exchange and execution across frameworks and runtimes.
• parameteruses · A model parameter is a numerical value, such as a weight or bias, that is learned or set to determine how a model computes outputs.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with R, Third Edition. The section links point to the parts of the book it draws on.