Data modality
A data modality is a distinct form or channel in which information is represented or conveyed, such as text, images, audio, or structured data.
Effective Conversational AI
The communication mode through which a user interacts with a conversational system, such as text, voice, email, or push notification.
- How it is used
- Guides virtual-assistant flow design around how users communicate and what their situations demand.
- How deep it goes
- Defines four interaction modes and compares their strengths, limits, and design implications without giving implementation code.
- Examples
- Contrasts text and video printer guidance, then poor and improved voice interactions for flight status.
- What you can do after
- Evaluate/understand it by choosing a suitable interaction mode and spotting avoidable friction in an existing assistant flow.
Read how Effective Conversational AI explains data modality →
The Art of AI Product Development
A modality is a distinct type or form of data that an AI system can process or generate.
- How it is used
- Maps raw data to appropriate AI techniques during discovery, solution selection, and user-experience design.
- How deep it goes
- Defines modalities, explains numeric transformation and multimodal combination, but treats individual modalities only through brief task examples.
- Examples
- Uses sentiment analysis, translation, summarization, question answering, image classification, object detection, and image generation.
- What you can do after
- Evaluate a product problem’s data modality, select relevant AI techniques, and recognize when combining modalities may fit.
Read how The Art of AI Product Development explains data modality →
AI-Powered Search
A data modality is a distinct type or channel of information, such as text, images, audio, or behavioral signals.
- How it is used
- Extends search beyond text by using multiple information channels to improve matching, ranking, and relevance.
- How deep it goes
- Explains modality types, vector representation, and multimodal querying, with implementation examples for image and text search.
- Examples
- Uses DeLorean image search, superhero text queries, movie embeddings, metadata, behavior signals, and multimodal vector retrieval.
- What you can do after
- Apply it via multimodal embeddings and vector search to retrieve content from text, images, or combined queries.
Deep Learning with Python, Third Edition
A type of data such as text, images, or audio that can be represented as sequences for model processing.
- How it is used
- Frames non-text inputs and outputs as sequences so Transformer-based models can process multiple data types.
- How deep it goes
- Traces image patches through encoding, dimensional alignment, placeholder replacement, and soft-token insertion before deferring broader training details.
- Examples
- Uses image patches, Gemma image prompts, and generated descriptions of a snake wearing glasses in a chair and reading.
- What you can do after
- Apply it by framing a modality as a compatible sequence and combining that representation with text for Transformer processing.
Read how Deep Learning with Python, Third Edition explains data modality →
Fight Fraud with Machine Learning
Data modality is the form in which data is represented, such as structured, unstructured, or semi-structured data.
- How it is used
- Guides fraud analysis by identifying available evidence, suitable detection approaches, and deployment requirements.
- How deep it goes
- Classifies structured, unstructured, and semi-structured data, then applies the distinction to image-based document forgery.
- Examples
- Uses transaction fields, text and images, combined phishing evidence, and forged digital medical bills as examples.
- What you can do after
- Evaluate fraud data by representation type and select detection approaches aligned with its modality.
Read how Fight Fraud with Machine Learning explains data modality →
Also covered in
- Machines that ThinkIn early access (MEAP)
- How Large Language Models Work
- Introduction to Generative AI, Second Edition
- Hugging Face in Action
- Transformers in Action
- Generative AI in Action
- Build a Text-to-Image Generator (from Scratch)
- Data Analysis with LLMs
- Time Series Forecasting Using Foundation Models
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- embeddingAn embedding is a numerical vector representation of an item in a feature space, encoding selected properties or relationships for computational tasks such as comparison or retrieval.
- digital imageA digital image is a digital representation of visual content, commonly encoded as an arrangement of pixels or other graphical elements.
- transformer architectureA transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
Related concepts
- multimodalused by · A multimodal model is a model that processes, relates, or generates information in more than one data modality.
- user experiencecontrasts with · User experience is a person’s overall perceptions and responses resulting from interacting with or using a product, service, or system.
- voice interfaceincludes · A voice interface enables people to communicate with a system through spoken input, with responses delivered through speech or other modalities.
Questions about data modality
What is data modality?
A data modality is a distinct form or channel in which information is represented or conveyed, such as text, images, audio, or structured data.
Which Manning books explain data modality?
Effective Conversational AI, The Art of AI Product Development, AI-Powered Search, Deep Learning with Python, Third Edition and Fight Fraud with Machine Learning.
What should I know before learning data modality?
Manning books that teach data modality usually introduce language model, embedding, digital image and transformer architecture first.
What is the difference between data modality and user experience?
A data modality is a distinct form or channel in which information is represented or conveyed, such as text, images, audio, or structured data. User experience is a person’s overall perceptions and responses resulting from interacting with or using a product, service, or system.
What kinds of data modality are there?
Kinds of data modality include visual modality, text interaction and voice interface.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.




