1. PUI 98 Leveraging Human Capabilities in Perceptual Interfaces George G. Robertson Microsoft Research George G. Robertson Microsoft Research

2. PUI 98 Outline and Goal What are perceptual interfaces? Perceptive vs perceptual Perceptive vs perceptual Multimodal interfaces Multimodal interfaces Challenge: Do our interfaces work? How do we find out? How do we find out? Challenge: Broaden our scope Leverage other natural human capabilities Leverage other natural human capabilities What are perceptual interfaces? Perceptive vs perceptual Perceptive vs perceptual Multimodal interfaces Multimodal interfaces Challenge: Do our interfaces work? How do we find out? How do we find out? Challenge: Broaden our scope Leverage other natural human capabilities Leverage other natural human capabilities

3. PUI 98 Perceptive to Perceptual Perceptive UI: aware of user Input to computer: use human motor skills Input to computer: use human motor skills Multimodal UI: use communication skills We use multiple modalities to communicate We use multiple modalities to communicate Perceptual UI: use many human abilities Perception, cognition, motor, communication Perception, cognition, motor, communication Perceptive UI: aware of user Input to computer: use human motor skills Input to computer: use human motor skills Multimodal UI: use communication skills We use multiple modalities to communicate We use multiple modalities to communicate Perceptual UI: use many human abilities Perception, cognition, motor, communication Perception, cognition, motor, communication

4. PUI 98 What are Modalities? Human communication channels Sensations (hearing or seeing)

5. PUI 98 What are Multimodal Interfaces? Attempts to use human communication skills Attempts to use human communication skills Provide user with multiple modalities Provide user with multiple modalities May be simultaneous or not May be simultaneous or not Fusion vs Temporal Constraints Fusion vs Temporal Constraints Multiple styles of interaction Multiple styles of interaction Attempts to use human communication skills Attempts to use human communication skills Provide user with multiple modalities Provide user with multiple modalities May be simultaneous or not May be simultaneous or not Fusion vs Temporal Constraints Fusion vs Temporal Constraints Multiple styles of interaction Multiple styles of interaction

6. PUI 98 Examples Bolt, SIGGRAPH80 Put That There Put That There Speech and gestures used simultaneously Speech and gestures used simultaneously Bolt, SIGGRAPH80 Put That There Put That There Speech and gestures used simultaneously Speech and gestures used simultaneously

7. PUI 98 Put That There

8. PUI 98 Examples (continued) Buxton and Myers, CHI86 Two-handed input Two-handed input Cohen et al, CHI89 Direct manipulation and NL Direct manipulation and NL Hauptmann, CHI89 Speech and gestures Speech and gestures Buxton and Myers, CHI86 Two-handed input Two-handed input Cohen et al, CHI89 Direct manipulation and NL Direct manipulation and NL Hauptmann, CHI89 Speech and gestures Speech and gestures

9. PUI 98 Examples (continued) Bolt, UIST92 Two-handed gestures and Gaze Two-handed gestures and Gaze Blattner & Dannenberg, 1992 book Hanne: text & gestures (interaction styles) Hanne: text & gestures (interaction styles) Pausch: selection by multimodal input Pausch: selection by multimodal input Rudnicky: speech, gesture, keyboard Rudnicky: speech, gesture, keyboard Bier et al, SIGGRAPH93 Tool Glass; two-handed input Tool Glass; two-handed input Bolt, UIST92 Two-handed gestures and Gaze Two-handed gestures and Gaze Blattner & Dannenberg, 1992 book Hanne: text & gestures (interaction styles) Hanne: text & gestures (interaction styles) Pausch: selection by multimodal input Pausch: selection by multimodal input Rudnicky: speech, gesture, keyboard Rudnicky: speech, gesture, keyboard Bier et al, SIGGRAPH93 Tool Glass; two-handed input Tool Glass; two-handed input

10. PUI 98 Examples (continued) Balboa & Coutaz, Intelligent UI93 Taxonomy and evaluation of MMUI Taxonomy and evaluation of MMUI Walker, CHI94 Facial expression (multimodal output) Facial expression (multimodal output) Nigay & Coutaz, CHI95 Architecture for fused multimodal input Architecture for fused multimodal input Balboa & Coutaz, Intelligent UI93 Taxonomy and evaluation of MMUI Taxonomy and evaluation of MMUI Walker, CHI94 Facial expression (multimodal output) Facial expression (multimodal output) Nigay & Coutaz, CHI95 Architecture for fused multimodal input Architecture for fused multimodal input

11. PUI 98 Why Multimodal Interfaces? Now fall far short of human capabilities Higher bandwidth is possible Higher bandwidth is possible Different modalities excel at different tasks Different modalities excel at different tasks Errors and disfluencies reduced Errors and disfluencies reduced Multimodal interfaces are more engaging Now fall far short of human capabilities Higher bandwidth is possible Higher bandwidth is possible Different modalities excel at different tasks Different modalities excel at different tasks Errors and disfluencies reduced Errors and disfluencies reduced Multimodal interfaces are more engaging

12. PUI 98 Leverage Human Capabilities Leverage senses and perceptual system Users perceive multiple things at once Users perceive multiple things at once Leverage senses and perceptual system Users perceive multiple things at once Users perceive multiple things at once Leverage motor and effector capabilities Leverage motor and effector capabilities Users do multiple things at once Users do multiple things at once

13. PUI 98 Senses and Perception Use more of users senses Not just vision Not just vision Sound Sound Tactile feedback Tactile feedback Taste and smell (maybe in the future) Taste and smell (maybe in the future) Users perceive multiple things at once e.g., vision and sound e.g., vision and sound Use more of users senses Not just vision Not just vision Sound Sound Tactile feedback Tactile feedback Taste and smell (maybe in the future) Taste and smell (maybe in the future) Users perceive multiple things at once e.g., vision and sound e.g., vision and sound

14. PUI 98 Motor & Effector Capabilities Currently: pointing or typing Much more is possible: Gesture input Gesture input Two-handed input Two-handed input Speech and NL Speech and NL Body position, orientation, and gaze Body position, orientation, and gaze Users do multiple things at once e.g., speak and use hand gestures e.g., speak and use hand gestures Currently: pointing or typing Much more is possible: Gesture input Gesture input Two-handed input Two-handed input Speech and NL Speech and NL Body position, orientation, and gaze Body position, orientation, and gaze Users do multiple things at once e.g., speak and use hand gestures e.g., speak and use hand gestures

15. PUI 98 Simultaneous Modalities? Single modality at a time Adapt to display characteristics Adapt to display characteristics Let user determine input mode Let user determine input mode Redundant, but only one at a time Redundant, but only one at a time Multiple simultaneous modalities Two-handed input Two-handed input Speech and hand gestures Speech and hand gestures Graphics and sound Graphics and sound Single modality at a time Adapt to display characteristics Adapt to display characteristics Let user determine input mode Let user determine input mode Redundant, but only one at a time Redundant, but only one at a time Multiple simultaneous modalities Two-handed input Two-handed input Speech and hand gestures Speech and hand gestures Graphics and sound Graphics and sound

16. PUI 98 Taxonomy (Balboa, 1993) Fusion TemporalConstraints Synergetic Exclusive IndependentSequentialConcurrent Put that there click … click Put that click there click multiple menu selection or multiple spoken commands Shortcuts

17. PUI 98 Modality = Style of Interaction Many styles exist Command interface Command interface NL NL Direct manipulation (WIMP and non-WIMP) Direct manipulation (WIMP and non-WIMP) Conversational (with an interface agent) Conversational (with an interface agent) Collaborative Collaborative Mixed styles produce multimodal UI Direct manipulation and conversational agent Direct manipulation and conversational agent Many styles exist Command interface Command interface NL NL Direct manipulation (WIMP and non-WIMP) Direct manipulation (WIMP and non-WIMP) Conversational (with an interface agent) Conversational (with an interface agent) Collaborative Collaborative Mixed styles produce multimodal UI Direct manipulation and conversational agent Direct manipulation and conversational agent

18. PUI 98 Multimodal versus Multimedia Multimedia is about media channels Text, graphics, animation, video: all visual media Text, graphics, animation, video: all visual media Multimodal is about sensory modalities Visual, auditory, tactile, … Visual, auditory, tactile, … Multimedia is a subset of Multimodal Output Multimedia is a subset of Multimodal Output Multimedia is about media channels Text, graphics, animation, video: all visual media Text, graphics, animation, video: all visual media Multimodal is about sensory modalities Visual, auditory, tactile, … Visual, auditory, tactile, … Multimedia is a subset of Multimodal Output Multimedia is a subset of Multimodal Output

19. PUI 98 Perceptual UI How Do The Pieces Fit? Multimodal Input Multimodal Output Multimedia Perceptive UI

20. PUI 98 Challenge Do our interfaces actually work? How do we find out? How do we find out? Do our interfaces actually work? How do we find out? How do we find out?

21. PUI 98 Why Test For Usability? Commercial efforts require proof Cost benefit analysis before investment Cost benefit analysis before investment Intuitions are great for design But intuition is not always right! But intuition is not always right! Peripheral Lens Peripheral Lens Commercial efforts require proof Cost benefit analysis before investment Cost benefit analysis before investment Intuitions are great for design But intuition is not always right! But intuition is not always right! Peripheral Lens Peripheral Lens

22. PUI 98 Peripheral Vision Does peripheral vision make navigation easier? Can we simulate peripheral vision? Does peripheral vision make navigation easier? Can we simulate peripheral vision?

23. PUI 98 A Virtual Hallway

24. PUI 98 Peripheral Lenses

25. PUI 98 Peripheral Lens

26. PUI 98 Peripheral Lens Intuitions Locomotion should be easier Especially around corners Especially around corners Wayfinding should be easier You can see far sooner You can see far sooner Locomotion should be easier Especially around corners Especially around corners Wayfinding should be easier You can see far sooner You can see far sooner

27. PUI 98 Peripheral Lens Findings Lenses were about the same speed Harder to use for inexperienced people Harder to use for inexperienced people Corner turning was not faster Corner turning was not faster Lenses were about the same speed Harder to use for inexperienced people Harder to use for inexperienced people Corner turning was not faster Corner turning was not faster

28. PUI 98 The Lesson Do not rely solely on intuition Test for usability! Do not rely solely on intuition Test for usability!

29. PUI 98 Challenge Are we fully using human capabilities? Peceptive UI is aware of the body Peceptive UI is aware of the body Multimodal UI is aware the we use multiple modalities, sometimes simultaneous Multimodal UI is aware the we use multiple modalities, sometimes simultaneous Perceptual UI should go beyond both of these Perceptual UI should go beyond both of these Are we fully using human capabilities? Peceptive UI is aware of the body Peceptive UI is aware of the body Multimodal UI is aware the we use multiple modalities, sometimes simultaneous Multimodal UI is aware the we use multiple modalities, sometimes simultaneous Perceptual UI should go beyond both of these Perceptual UI should go beyond both of these

30. PUI 98 Research Strategy ExploitTechnologyDiscontinuities Compelling Task: InformationAccess LeverageHumanCapabilities

31. PUI 98 communication Engaging Human Abilities understand complexity new classes of tasks less effort understand complexity new classes of tasks less effort HelpsUser perceptual motor cognitive

32. PUI 98 Examples: Communication LanguageGestureAwarenessEmotionMultimodal Flexible Flexible Robust Robust Dialogue to resolve ambiguity Dialogue to resolve ambiguity

33. PUI 98 Examples: Communication LanguageGestureAwarenessEmotionMultimodal Hands Hands Body pose Body pose Facial expression Facial expression

34. PUI 98 Camera-Based Conversational Interfaces Leverage face to face communication skills

35. PUI 98 Examples: Communication LanguageGestureAwarenessEmotionMultimodal Is anybody there? Is anybody there? Doing what? Doing what?

36. PUI 98 Camera-Based Awareness What is the user doing?

37. PUI 98 Examples: Communication LanguageGestureAwarenessEmotionMultimodal Social response Social response Perceived personality Perceived personality

38. PUI 98 Examples: Communication LanguageGestureAwarenessEmotionMultimodal Natural Natural Choice Choice Reduces errors Reduces errors Higher bandwidth Higher bandwidth

39. PUI 98 Examples: Motor Skills Bimanual skills Muscle memory Bimanual skills Muscle memory Multimodal Map Manipulation Two hands Two hands Speech Speech

40. PUI 98 Camera-Based Navigation How do our bodies move when we navigate?

41. PUI 98 Examples: Perception Spatial relationships Pattern recognition Object constancy Parallax Other Senses Spatial relationships Pattern recognition Object constancy Parallax Other Senses Cone Tree Xerox PARC Information Visualizer

42. PUI 98 Cone Tree

43. PUI 98 Examples: Perception Spatial relationships Pattern recognition Object constancy Parallax Other Senses Spatial relationships Pattern recognition Object constancy Parallax Other Senses Key 3D depth cue Key 3D depth cue Sensor issues Sensor issues Camera-based head- motion parallax Camera-based head- motion parallax

44. PUI 98 Camera-Based Head-Motion Parallax Motion parallax is one of strongest 3D depth cues

45. PUI 98 Examples: Perception Spatial relationships Pattern recognition Object constancy Parallax Other Senses Spatial relationships Pattern recognition Object constancy Parallax Other Senses Auditory Auditory Tactile Tactile Kinesthetic Kinesthetic Vestibular Vestibular Taste Taste Olfactory Olfactory

46. PUI 98 Examples: Perception Olfactory? Maybe soon? Ferris Productions Olfactory VR Add-on Time, April 29, 1996 Barfield & Danas Olfactory Displays Presence, Winter, 1995

47. PUI 98 Examples: Cognition Spatial memory Cognitive chunking AttentionCuriosity Time Constants Data Mountain

48. PUI 98 Data Mountain Favorites Management Exploits: Spatial memory Spatial memory 3D perception 3D perception Pattern recognition Pattern recognition Favorites Management Exploits: Spatial memory Spatial memory 3D perception 3D perception Pattern recognition Pattern recognition Advantages: Spatial organization Spatial organization Not page at a time Not page at a time 3D advantages with 2D interaction 3D advantages with 2D interaction

49. PUI 98 Sample User Reaction Subject Layout of 100 Pages Strongest cue... relative size

50. PUI 98 VIDEO

51. PUI 98 Data Mountain Usability Spatial memory works in virtual environments! 26% faster than IE4 Favorites 2x faster with Implicit Query Spatial memory works in virtual environments! 26% faster than IE4 Favorites 2x faster with Implicit Query

52. PUI 98 Implicit Query Visualization Highlight related pages Slightly slower for storage Over 2x faster for retrieval Highlight related pages Slightly slower for storage Over 2x faster for retrieval

53. PUI 98 Examples: Cognition Spatial memory Cognitive chunking AttentionCuriosity Time Constants Navigate Map ZoomPan dXdYfactorCenter XY

54. PUI 98 Examples: Cognition Spatial memory Cognitive chunking AttentionCuriosity Time Constants Motion attracts Motion attracts Animate with care Animate with care Peripheral vision Peripheral vision HMD vs desktop HMD vs desktop Focus in Context Focus in Context

55. PUI 98 Focus in Context

56. PUI 98 Examples: Cognition Spatial memory Cognitive chunking AttentionCuriosity Time Constants Discoverability Discoverability Fear Fear Universal Undo Universal Undo

57. PUI 98 Examples: Cognition Spatial memory Cognitive chunking AttentionCuriosity Time Constants Unit Cognitive Task Animation Immediate Response (sec) 0.1 1 10 100

58. PUI 98 Summary: Recommendations Broaden scope! Identify and engage human abilities Identify and engage human abilities Go beyond the perceptive and multimodal Go beyond the perceptive and multimodal Test for usability! Broaden scope! Identify and engage human abilities Identify and engage human abilities Go beyond the perceptive and multimodal Go beyond the perceptive and multimodal Test for usability!