Multimodal data combines information from multiple modalities, such as text, images, audio, or video.
Multimodal data combines different kinds of input, such as text and images. A language-model prompt can pair a question with an image so the model can answer about what it sees. section 6.2.1section 1.3.1
Ask a question about an image
A prompt pairs an image with the question “Does this picture show a banana?” The expected answer is “No.” section 6.2.1
Figure 6.1 Multimodal prompt containing an image and text. The prompt instructs the language model to decide whether the picture shows a banana. In this case, the expected output is “No” (otherwise “Yes”).
See the image-and-text prompt example for a visual question-answering task. section 6.2.1
How Data Analysis with LLMs covers it
How deep it goes
Explains how a message represents text and images as typed elements.
Examples
Pairs an image with a banana-presence question and describes generating a poem and picture from instructions.
What you can do after
Build a user message containing a text question and image URL for visual question answering.
How it connects
Connects prompts, language models, text, images, and visual question answering.
GPT-4oused by · GPT-4o is an OpenAI multimodal generative model in the GPT family that processes and generates text and other supported modalities.
multimodalused by · A multimodal model is a model that processes, generates, or relates information represented in two or more data modalities, such as text, images, audio, or video.
promptused by · An input, typically containing instructions or context, supplied to an artificial-intelligence model to guide or elicit its output.
datakind of · Data is represented information, observations, measurements, or other values that can be stored, processed, transmitted, analyzed, or interpreted.
outputcontrasts with · An output is a result or response produced by a process, operation, or system, including data, content, actions, or signals.
This explainer was generated by Manning's knowledge graph from the text of Data Analysis with LLMs. The section links point to the parts of the book it draws on.
