Multimodal interaction combines channels such as text, voice, images, and video. It can make an assistant more versatile, but combining channels adds design complexity. Choose modalities around user needs and the strengths of each channel. section 9.2 section 9.2.1
A spoken question with visual results
A user asks a question aloud; the assistant displays relevant photos and replies in both text and voice. A voice bot might also offer a video tutorial or other visual aid. section 9.2.1 section 9.2.4
Table 9.3 Modality overview
|
Text modality
|
Visual modality
|
Voice modality
|
Multimodal interactions
|
|
|---|---|---|---|---|
|
Description
|
Communication through written text
|
Communication using text with additional visual elements
|
Interaction via spoken language
|
Combines text, voice, and visual elements
|
|
Strengths
|
Precise communication.
Easy to reference.
|
Enhances comprehension with visuals through an engaging interface
|
Hands-free operation.
Natural conversational flow.
|
Richer user experience.
Versatile interaction.
|
|
Challenges
|
Typing can be slow.
May lack emotional tone.
|
Visuals may distract or be tuned out.
Requires a well-designed UI.
|
Requires clear speech.
Can misinterpret accents.
|
More complex to design and implement
|
|
When to use
|
In situations where typing is convenient
|
Scenarios needing visual aid for better understanding
|
Situations requiring hands-free interaction
|
Scenarios needing a combination of modalities
|
|
Effect on user engagement
|
High engagement in text-centric contexts
|
Improved engagement with visual aids
|
Increased engagement due to natural interaction
|
Enhanced engagement through diverse interaction modes
|
|
Evaluation techniques
|
User feedback surveys.
Usability testing.
|
User experience testing.
Visual effectiveness analysis.
|
Speech recognition accuracy tests.
User feedback.
|
A/B testing.
Multimodal usability assessments.
|
Compare text, visual, voice, and multimodal interactions by their strengths, challenges, and suitable contexts. section 9.2.1
