1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection

Outline Research problem Previous research Research questions Research strategy and focus Prototype Empirical studies Contributions Future work 2

Research problems and objectives Motivating problem – How can users interact with information devices in mobile environments when their visual attention is taken up by other tasks (e.g navigation through the environment) Objective – To develop interactive techniques that support user control and feedback without relying on the visual modality Scope – This research focuses on menu selection because this is an elemental task (Foley et al. ‘84) 3

Previous research Visual menus – Linear menus (e.g., Sear and Schneiderman ’94; Cockburn and Gin ‘06, etc.) – Radial menus (e.g., Callahan et al. ‘88; Kurtenbach ’93, etc.) Auditory applications – Accessibility for the blind (e.g., Edwards ‘89; Mynatt and Weber ‘94, etc.) – Mobile auditory interfaces (e.g., Roy and Schmandt ‘96; Pirhonen et al. ’02, etc.) Auditory menus – Interactive Voice Response (IVR) systems (e.g., Marics and Engelbeck 1997; Resnick and Virzi ’92, etc.) – Head gesture auditory pie menu (Brewster et al. ‘03) 4

Research questions How to design usable eyes-free menus? – Designing interfaces with reasonable levels of efficiency and ease of use – Facilitating a smooth novice to expert transition How to work around the problems with serial and temporal audio feedback? – Facilitating scan and compare of menu items – Providing users with a sense of control and responsiveness 5

Research strategy Define research problem Review literature Define general design concept Perform iteratively design and pilot studies for prototype Conduct empirical studies to explore prototype properties Derive design recommendations based on empirical results 6

Selected input and output modalities Touch input – Combines relative and absolute pointing (Hinckley ‘01) – Can be robustly operated in motion (Buxton et al. ‘85) – Can be combined with other input devices (Hinckley and Sinclair ‘99; Rekimoto et al. ‘03) Auditory feedback – The other primary remote sense (Gaver ‘97) – The only other sense with a natural language* 7

earPod prototype A touch-sensitive device with software to perform eyes-free, hierarchical menu selection earPod prototype used commercial touchpad augmented with custom-built plastic cover, with cable to computer for voice output Built in University of Toronto in 2006 with Pierre Dragicevic 8

earPod Prototype 9

earPod interaction 10 NoviceIntermediate Expert

Video - browsing 11

Video - fast browsing 12

Video - very fast browsing 13

Video - expert use 14

Prototype design Efficiency – By-pass audio – Use interruptible audio feedback – Use non-speech audio Sense of control – Synchronous communication (reactive audio feedback) Simplicity and power – Gliding and tapping Easy to learn – Use spatial audio to reinforce mental model – Self-discoverable transition 15

Empirical studies Motivating questions: 1.How does earPod compare with a popular visual menu selection technique used by iPod? 2.How does earPod compare with a number of competitor techniques with different modalities and menu styles? 3.What are learning behaviors for earPod and the related techniques? 4.How does earPod and the related techniques perform with a visually demanding primary task? 16

Evaluation issues Modality – Visual – Audio – Dual Menu style – Linear – Radial 17 Menu Style Radial (absolute) Linear (relative) Modality Audio Audio radial Audio linear Visual Visual radial Visual linear Dual Audio visual radial Audio visual linear

4 empirical studies 4 empirical studies were conducted at UofT in Experiment 1: earPod vs. iPod (audio radial vs. visual linear) Experiment 2: 3 x 2 study (3 modalities x 2 menu styles) Experiment 3: 2 x 2 longitudinal study (audio linear, visual linear, audio radial, visual radial) Experiment 4: 3 x 2 dual-task study (3 modalities x 2 menu styles) 18

Experimental 1 design Goal: compare earPod with iPod-like linear menu minute session, within-subject, counter- balanced Setup: 12 participants x 2 techniques (audio and visual) x (40+80) items for the 2 menu configurations (8 and 8x8) x 4 blocks = 11,520 menu selections in total. 19

Results: response time Block number 20

Results: response time Block number 21

Results: response time Block number 22

Experiment 1 - results and discussion earPod has comparable performance with iPod-like linear menu earPod user performance improves more rapidly than that of iPod-like linear menu With practice, earPod can outperform iPod- like linear menu 23

Experiment 2 design Goal: systematic investigation of modality of feedback and menu style 60-minute, within subject, counter-balanced Setup 12 participants x 6 techniques x 8 menu items x 13 blocks (12 blocks + 1 practice block) = 7488 menu selections in total 24

25 Response time AudioradialAudiolinearVisualradialVisuallinear Audio visual radial linear

Experiment 2 - results and discussion Accuracy – Comparable performance among all techniques Speed – visual radial ~ dual radial < audio radial ~ visual linear ~ dual linear < audio linear Modality – Visual ~ dual < audio Menu style – Radial < linear 26

Experiment 3: longitudinal study Goal: learning of earPod and related techniques 5 one-hour sessions in a week, between-subject Setup: 2 participants (per technique) x 4 techniques x 64 items for 1 menu configuration (8 x 8) x 7 blocks per day x 5 days = menu selections in total 27

Accuracy 28 Day

Response time 29 Day

Gliding to tapping transition 30 Items exploredTapping percentage Day

Experiment 3 - results and discussion Novice performance is determined by modality, expert performance is determined by menu style Rapid learning is observed for earPod Users can perform tapping for 70-80% of the 64 menu items from day 3 onwards Audio linear has comparable performance with visual linear from day 3 onwards 31

Experiment 4: dual-task study Goal: Investigate the properties of using earPod and the related techniques as a secondary task in a dual-task setting and their impact on the visually demanding primary task 32

Experiment 4: dual-task study Dual-task – Primary task: simulated driving – Secondary task: menu selection External validity: similar settings have been successfully used by Salvucci et al. ‘01, ‘02, ‘05 33

Experiment 4 design Conditions – Desktop 12 participants X 6 techniques X 8 items of 1 menu configurations X (3 + 1) blocks = 2304 menu selections – Driving 12 participants X 6 techniques X 8 items of 1 menu configurations X (1 + 1) blocks = 1152 menu selections Measures – Desktop Speed & accuracy of menu selection – Driving Speed & accuracy of menu selection (secondary task) Driving performance (primary task) 34

Experiment 4 setup 35

Response time: modality 36 AudioVisualDual

Response time: menu style 37 P RadialLinear

Lateral velocity 38

Following distance 39

Experiment 4 - results and discussion Audio techniques work better with a visually demanding primary task – Comparable performance with visual techniques, less impact on primary tasks Although dual-channel techniques have similar performance with audio techniques, users prefer single channel audio feedback 40

Summary of empirical studies earPod has comparable performance with visual linear menus Transition from novice to expert for earPod is fast Audio techniques work better with a visually demanding primary task Users only pay attention to the appropriate channel when are presented with both channels of feedback 41

Contributions Developed a novel interaction technique called earPod Conducted empirical evaluations on earPod and related techniques Derived design recommendation for incorporating earPod and related techniques to mobile devices 42

Future work Explore eyes-free menu techniques for long and dynamic lists Investigate the role of compression and dichotic listening Extend earPod to other mobile scenarios (e.g., walking, running, etc.) Develop eyes-free interaction techniques for other tasks and applications (e.g., drag & drop, text entry, etc.) 43

Questions 44

45 The End