A data modality is a distinct type or form of data that an AI system can process or generate. Identifying it early helps align data with user needs, model objectives, and suitable AI techniques. section 3.1 section 3.1.1
Common raw-data modalities support different tasks and technical approaches:
- Textual data supports natural language processing tasks such as sentiment analysis, translation, summarization, and question answering. section 3.1.1
- Visual data includes images and videos; computer vision supports tasks such as image classification, object detection, and image generation. section 3.1.1
- Auditory data forms another central modality alongside text and visual data. section 3.1.1
Although raw data can be textual, visual, or auditory, models require numeric input, so the data must be transformed into a numerical representation. Multimodal AI combines multiple modalities and is becoming more accessible through multimodal foundational models. section 3.1.1 section 3.1.1
Modality choices influence discovery, team expertise, model selection, and user experience. They also connect with learning setup: labeled data supplies clear signals for supervised learning, while unlabeled data supports unsupervised learning. section 3.1 section 3.4
Use the modality relationship figure to compare single-modality and multimodal data choices. section 3.1.1
