1. Multimodal Interfaces
Scott Klemmer
09 November 2004

2. Multimodal Interfaces
Some hci definitions
Multimodal generally refers to an interface that can accept input from two or more combined modes
Multimedia generally refers to an interface that produces output in two or more modes
The vast majority of multimodal systems have been speech + pointing (pen or mouse) input, with graphical (and sometimes voice) output
09 November 2004
Multimodal Interfaces

3. Multimodal Interfaces
Canonical App: Maps
Why are maps so well-suited?
A visual artifact for computation (Hutchins)
09 November 2004
Multimodal Interfaces

4. Multimodal Interfaces
What is an interface
Is it an interface if there’s no method for a user to tell if they’ve done something?
What might an example be?
Is it an interface if there’s no method for explicit user input?
example: health monitoring apps
09 November 2004
Multimodal Interfaces

5. Multimodal Interfaces
Sensor Fusion
multimodal = multiple human channels
sensor fusion = multiple sensor channels
Example app: Tracking people (1 human channel)
might use: RFID + vision + keyboard activity + …
I disagree with the Oviatt paper
Speech + lips is sensor fusion, not multimodality
09 November 2004
Multimodal Interfaces

6. What constitutes a modality?
To some extent, it’s a matter of semantics
Is pen a different modality than a mouse?
Are two mice different modalities if one is controlling a gui, and the other controls a tablet-like ui?
Is a captured modality the same as an input modality?
How does the audio notebook fit into this?
09 November 2004
Multimodal Interfaces

7. Multimodal Interfaces
Input modalities
mouse
pen: recognized or unrecognized
speech
non-speech audio
tangible object manipulation
gaze, posture, body-tracking
Each of these experiences has different implementing technologies
e.g., gaze tracking could be laser-based or vision-based
09 November 2004
Multimodal Interfaces

8. Multimodal Interfaces
Output modalities
Visual displays
Raster graphics, Oscilloscope, paper printer, …
Haptics: Force Feedback
Audio
Smell
Taste
09 November 2004
Multimodal Interfaces

9. Multimodal Interfaces
Dual Purpose Speech
09 November 2004
Multimodal Interfaces

10. Multimodal Interfaces
Why multimodal?
Hands busy / eyes busy
Mutual disambiguation
Faster input
“More natural”
09 November 2004
Multimodal Interfaces

11. Multimodal Interfaces
On Anthropomorphism
The multimodal community grew out of the AI and speech communities
Should human communication with computers be as similar as possible to human-human communication?
09 November 2004
Multimodal Interfaces

12. Multimodal Software Architectures
OAA, AAA, OOPS
09 November 2004
Multimodal Interfaces