Visual encoder

As explained in Deep Learning with Python, Third Edition

A model component that converts images into vector representations suitable for a larger multimodal model.

A visual encoder converts an image into a sequence of vectors that a larger multimodal model can process alongside text. This works because Transformers learn patterns in sequences, not only text. section 16.4.2

Turning an image into soft tokens

Gemma’s 420-million-parameter image encoder cuts an image into 256 patches and encodes each patch as a 2,560-dimensional vector. The resulting (256, 2560) sequence matches Gemma’s hidden dimension and can be spliced into the text sequence after token embedding. The special token <start_of_image> becomes 256 placeholders, which are replaced by these image vectors. Unlike fixed-ID hard tokens, soft tokens can take any vector value produced by the encoder. section 16.4.2

Use the encoder according to the surrounding model architecture:

• For multimodal generation, align the encoder output dimension with the Transformer hidden dimension so image vectors can join the text sequence. section 16.4.2
• For segmentation, pass the smaller image embedding to a mask decoder together with a prompt embedding; the encoder alone does not produce the final mask. section 11.3.2
• Treat image encoding as a separable training stage when preparing a multimodal model, then train on mixed image-and-text sequences. section 16.4.2

Figure 16.6 Handling image input by splicing text tokens and soft image tokens together

Shows how image soft tokens and text tokens occupy one Transformer input sequence. section 16.4.2

How Deep Learning with Python, Third Edition covers it

How deep it goes

Explains patch extraction, vector dimensions, sequence insertion, and training placement, while deferring encoder internals.

Examples

Uses Gemma image questions and Segment Anything’s image-prompt-mask architecture as concrete applications.

What it assumes

Assumes basic familiarity with images, vectors, token embeddings, and Transformer sequences while treating multimodal conversion conceptually.

What you can do after

Operate/configure Gemma for image questions and understand how encoded images enter a multimodal Transformer.

How it connects

Connects image patches with Transformers, token embeddings, soft tokens, image embeddings, prompt encoders, and mask decoders.

liveBookmanning.com

Read Deep Learning with Python, Third Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Deep Learning with R, Third EditionAn image encoder is a model component that transforms an input image into a lower-dimensional image embedding.
• 

  liveBookmanning.com
  Build a Text-to-Image Generator (from Scratch)An image encoder is a neural-network component that maps an image to a numerical representation.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionThe part of a multimodal model that converts an image into a numeric representation for downstream decoding.

Compare how 4 books explain visual encoder →

Related concepts

• CLIPpart of · CLIP is a multimodal model and pretraining method that aligns image and text representations in a shared embedding space.
• Gemmapart of · Gemma is a family of open-weight generative AI models developed by Google.
• SAMpart of · SAM, or Segment Anything Model, is a pretrained promptable computer-vision model that generates segmentation masks for objects in images.
• ONNXproduces · ONNX is an open standard for representing machine-learning models in a portable format that supports exchange and execution across frameworks and runtimes.
• parameteruses · A model parameter is a numerical value, such as a weight or bias, that is learned or set to determine how a model computes outputs.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with Python, Third Edition. The section links point to the parts of the book it draws on.