Multimodal

As explained in Machine Learning System Design

A property of a system or model that represents or processes information from multiple data modalities, such as text and images.

Multimodal describes data or systems that combine different kinds of information to describe one problem. A modality is a kind of information received, such as an image or text description. section 6.1

Combining modalities can support practical decisions when one dataset lacks necessary context:

• Match product images with descriptions to represent items more completely. section 6.1
• Combine satellite imagery with crop and yield information to improve agricultural operating efficiency. section 6.1
• Join paying-customer information with behavioral user profiles to target similar potential customers. section 6.1

Figure 6.1 Combining data sources of two completely different businesses can eventually benefit both.

Use the figure to visualize how data sources from different businesses can be combined for mutual benefit. section 6.1

How Machine Learning System Design covers it

How deep it goes

Defines modality and multimodal datasets through examples, but stops short of modeling, fusion, or implementation details.

Examples

Uses product images with descriptions, satellite imagery with agricultural records, and customer data with behavioral profiles.

What it assumes

Teaches modality from a basic definition and assumes only familiarity with combining datasets from different origins.

What you can do after

Apply multimodal data combinations to add missing context and identify opportunities for marketing or operational efficiency.

How it connects

Connects datasets, data sources, images, text descriptions, behavioral profiles, and online marketing.

liveBookmanning.com

Read Machine Learning System Design on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Introduction to Generative AI, Second EditionA multimodal interface supports interaction through multiple data modalities, such as text, images, audio, speech, or code.
• 

  liveBookmanning.com
  The Art of AI Product DevelopmentA multimodal model processes or generates information across multiple modalities such as text, images, audio, or video.
• 

  liveBookmanning.com
  Transformers in ActionA multimodal model is a model that processes or generates information across multiple data modalities such as text, audio, video, and images.
• 

  liveBookmanning.com
  LLMs in ProductionA multimodal solution is an AI system or method that processes or combines information from multiple data modalities.
• 

  liveBookmanning.com
  Generative AI in ActionA multimodal model is a model that processes or generates information in multiple modalities such as text, images, audio, or video.
• 

  liveBookmanning.com
  Hugging Face in ActionA multimodal model is a machine learning model that jointly processes or relates inputs from multiple data modalities, such as images and text.
• 

  liveBookmanning.com
  Build a Text-to-Image Generator (from Scratch)A generative model that processes or connects two or more data modalities, such as text, images, audio, or video.
• 

  liveBookmanning.com
  AI-Powered SearchA generative multimodal model generates content while accepting or representing information from multiple data modalities.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorchA generative model that accepts more than one data modality, such as text plus images or audio, as input.
• 

  liveBookmanning.com
  Build AI-Enhanced Web AppsAI technology that processes and generates content across multiple input or output modalities, such as text, images, and audio.

Compare how 11 books explain multimodal →

Related concepts

• language modelkind of · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
• CLIPincludes · CLIP is a multimodal model and pretraining method that aligns image and text representations in a shared embedding space.
• GPT-4includes · GPT-4 is a generative large language model developed by OpenAI for understanding and generating text and other supported inputs and outputs.
• data modalityuses · A data modality is a distinct form or channel in which information is represented, conveyed, or processed, such as text, images, or audio.
• transformer architectureuses · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning System Design. The section links point to the parts of the book it draws on.