CLIP

As explained in Learn Generative AI with PyTorch

A multimodal Transformer model that learns aligned image and text representations for comparing and matching visual and textual inputs.

CLIP is a multimodal Transformer that connects images and natural-language descriptions. Separate image and text encoders project both inputs into a shared embedding space, making matching visual and textual content comparable. section 15.0 section 15.5.1

Unlike models that depend on explicit image labels, CLIP learns from large collections of image–description pairs. Its contrastive objective increases similarity for matching pairs and decreases similarity for mismatched pairs. section 15.5.1

A CLIP training pass follows these ideas:

• Preprocess images to a common shape and tokenize their descriptions. section 15.5.1
• Encode each image with a Vision Transformer and each description with a Transformer-based language model. section 15.0
• Project both modalities into D-dimensional embeddings that can be compared. section 15.5.1
• For a batch of N pairs, maximize matching similarities on the diagonal and minimize nonmatching similarities off the diagonal. section 15.5.1

Figure 15.8 How OpenAI’s CLIP model is trained. A large-scale training dataset of text–image pairs is collected. The text encoder of the model compresses the text description into a D-value text embedding. The image encoder converts the corresponding image into an image embedding also with D values. During training, a batch of N text–image pairs are converted to N text embeddings and N image embeddings. CLIP uses a contrastive learning approach to maximize the similarity between paired embeddings (the sum of diagonal values in the figure) while minimizing the similarity between embeddings from nonmatching text–image pairs (the sum of off-diagonal values in the figure).

The training diagram makes the dual encoders, shared embedding dimensions, and diagonal-versus-off-diagonal contrastive objective concrete. section 15.5.1

CLIP also supplies multimodal understanding for text-to-image systems. In DALL-E 2, text becomes an embedding, CLIP produces a prior representing an image in latent space, and a diffusion process uses the resulting conditioning information to denoise random noise into an image. section 15.5 section 15.5.1

From prompt to generated image

The prompt “an astronaut in a space suit riding a unicorn” can drive DALL-E 2 image generation. A Python call selects the model, prompt, size, quality, and number of outputs, then returns a temporary image URL. section 15.5.2

How Learn Generative AI with PyTorch covers it

How deep it goes

Explains dual encoders, preprocessing, embeddings, and batch contrastive learning, then connects CLIP to DALL-E 2 generation.

Examples

Uses paired image–description batches, a training-similarity diagram, and DALL-E 2 prompts such as an astronaut riding a unicorn.

What it assumes

Assumes basic familiarity with images, text descriptions, embeddings, and Transformers while defining the multimodal training workflow.

What you can do after

Apply it via DALL-E 2 to understand prompt conditioning and generate images from textual descriptions.

How it connects

Links Vision Transformers, Transformer language models, contrastive learning, diffusion models, and DALL-E 2.

liveBookmanning.com

Read Learn Generative AI with PyTorch on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Build a Text-to-Image Generator (from Scratch)Contrastive language-image pretraining is a training technique that aligns image and text representations by contrasting matching and nonmatching pairs.
• 

  liveBookmanning.com
  AI-Powered SearchCLIP is a multimodal model trained to align images and their associated text in a shared embedding space.
• 

  liveBookmanning.com
  Hugging Face in ActionCLIP is a multimodal model trained to align images and text in a shared representation space.

Compare how 4 books explain CLIP →

Related concepts

• multimodalkind of · A multimodal model is a model that processes, generates, or relates information represented in two or more data modalities, such as text, images, audio, or video.
• encoderhas part · An encoder is a component that transforms input data into an encoded representation for subsequent processing, reconstruction, comparison, or transmission.
• contrastive learninguses · Contrastive learning is a representation-learning technique that trains a model to bring representations of related examples closer and separate representations of unrelated examples.
• latent diffusion modelpart of · A latent diffusion model is a generative model that performs iterative denoising in a learned latent space to synthesize data such as images.
• text encoderhas part · A text encoder is a neural-network model that transforms textual input into numerical representations for conditioning or other downstream computation.
• embeddingproduces · A numerical vector representation of an item or input that encodes features or relationships useful for computational tasks.

This explainer was generated by Manning's knowledge graph from the text of Learn Generative AI with PyTorch. The section links point to the parts of the book it draws on.