1. University of Toronto earPod: Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback Shengdong Zhao Pierre Dragicevic Mark Chignell Ravin Balakrishnan Patrick Baudisch

4. Menu Selection on Mobile Devices Often Requires Visual Feedback

5. Why Eyes-free?

7. Eyes-free capable

8. Marking Menus

9. Eyes-free capable Marking Menus (Expert Use)

10. Eyes-free capable Marking Menus (Expert Use)

11. True eyes-free TouchPlayer (Pirhonen et al, 2002)

12. True eyes-free TouchPlayer (Pirhonen et al, 2002) Head gesture-based auditory pie menu (Brewster et al, 2003)

13. earPod 13

14. earPod 14

15. earPod Design 15

16. earPod Walkthrough 16

17. Browsing

18. Fast browsing 18

19. Fast browsing

20. Very fast browsing and cancel 20

21. Very fast browsing and cancel

22. Expert use

24. Interactive Voice Response System 24

25. Interactive Voice Response System 25 Slow Painful “ Touchtone Hell ” (Yin and Zhai, 2006)

26. Visual vs. auditory menu 26 Visual Linear Menu

27. Visual vs. auditory menu 27 Visual Linear Menu IVR System

28. Problems of IVR system Speech is serial and slow (Slowiaczek and Nusbaum 1985)

29. Problems of IVR system Synchronous vs. asynchronous communication Synchronous Visual Outputs Mouse Input time User System

30. Problems of IVR system Synchronous vs. asynchronous communication Synchronous Visual Outputs Mouse Input time User System Asynchronous time Audio Press Key User System Waiting Press Key Waiting Audio

31. Problems of IVR system Transition to expert use Skill development Skill level Novice (Voice menu: recognition)

32. Problems of IVR system Transition to expert use Skill development Skill level Expert (direct code access: recall) Novice (Voice menu: recognition)

33. Problems of IVR system Transition to expert use Skill development Skill level Expert (direct code access: recall) Novice (Voice menu: recognition)

34. earPod ’ s approach Problem: speech is serial and slow

35. earPod ’ s approach Problem: speech is serial and slow –By-pass audio

36. earPod ’ s approach Problem: speech is serial and slow –By-pass audio –Use interruptible audio feedback

37. earPod ’ s approach Problem: speech is serial and slow –By-pass audio –Use interruptible audio feedback –Use non-speech audio

38. earPod ’ s approach Problem: asynchronous communication

39. earPod ’ s approach Problem: asynchronous communication –Synchronous communication (reactive audio feedback) Audio Outputs Touchpad Input time User

40. earPod ’ s approach Problem: steep novice to expert transition

41. Learning menu structure Smooth transition

42. Learning menu structure Learning expert gesture Smooth transition Target Item

43. Learning menu structure Learning expert gesture Smooth transition Target ItemNovice

44. Learning menu structure Learning expert gesture Smooth transition Target ItemNoviceNovice/Expert

45. Learning menu structure Learning expert gesture Smooth transition Target ItemNoviceNovice/ExpertExpert

46. 46 Skill development Skill level Expert (direct tapping: recall) Novice (glide browsing: recognition)

47. Support Single-hand Usage 47 (Karlson et al, 2007)

48. Haptic guidance

50. earPod Design Summary Touch Input 50

51. earPod Design Summary Touch Input Reactive & Interruptible Audio 51

52. earPod Design Summary Touch Input Reactive & Interruptible Audio Non-speech Audio 52

53. earPod Design Summary Touch Input Reactive & Interruptible Audio Non-speech Audio Input/Output Mapping 53

54. earPod Design Summary Touch Input Reactive & Interruptible Audio Non-speech Audio Input/Output Mapping Direct Item Access 54

55. earPod Design Summary Touch Input Reactive & Interruptible Audio Non-speech Audio Input/Output Mapping Direct Item Access Transition to Expert Use 55

56. Experiment 56 earPod vs. iPod-like linear menu

57. Experiment – visual condition 57

58. Experiment – audio condition 58

59. Experimental design Within-subject, counter-balanced 12 participants x 2 techniques (audio and visual) x (40+80) items for the 2 menu configurations (8 and 8x8) x 4 blocks = 11,520 menu selections in total. 59

60. Results 60 Block number

61. Results 61 Block number

62. Results 62 Block number

63. Results 63 Block number

64. Results 64 Block number

65. Results Detailed Log Analysis –9/12 reached expert use (1 level menu) –8/12 reached expert use (2 level menu) 65

66. Results Detailed Log Analysis –9/12 reached expert use (1 level menu) –8/12 reached expert use (2 level menu) Subjective Feedback –4 preferred earPod –4 preferred iPod-like visual menu –4 had no preferences –11/12 suggest combine audio & visual 66

67. earPod limitations Work with structured menu of limited breadth 67

68. earPod limitations Work with structured menu of limited breadth 68

69. Conclusion earPod: eyes-free menu selection for structured menu of limited breadth Performance comparable to iPod- like linear menus Support single-handed interaction Design for mobile use 69

70. Future Work Long/dynamic menus 70

71. Future Work Long/dynamic menus Dual channel menu 71

72. Future Work Long/dynamic menus Dual channel menu Dichotic listening (Ranjan et al., 2006) 72

73. Future Work Long/dynamic menus Dual channel menu Dichotic listening (Ranjan et al., 2006) IVR on the phone 73

74. Acknowledgement Danielle Lottridge, Tomer Moscovich, Mike McGuffin, Jingnan Yang, Heather Thorne, Alvin Chin, Derek Nowrouzezahrai, Yongchang Zhao, Lisa Min, Andrew Chignell, Paula Gilverson, Michelle Qu, Andrew McGutman, Noah Lockwood, John Chattoe, Annie Xu, Silvia Gonzaloz, Anna Malandrino Anonymous reviewers

75. Mobile Usage Design Stability issues 75

76. earPod Novice to Expert Transition 2 6 17 35 0 4