1. designing easily learnable eyes-free interaction Kevin Li University of California, San Diego

2. can we make easily learnable eyes-free interfaces by leveraging existing learned associations?

3. touch mapping eyes free interaction humanobject hearing musicspeechsounds Tapping and Rubbing Simulating buttons People Tones People Tones Mapping language to tactile stimulus class sense example project People Tones People Tones lessons for application designers

4. why eyes free?

5. PCs… PC screens have the users’ undivided attention  design for the visual channel

6. environment

7. screen-less device

8. can’t see screen

9. visual impairment

10. socially unacceptable

11. easily learnable

12. auditory feedback

13. calendar preview “Monday 9am” “tic, tic, sssssh” “How about Monday morning?” “Yeah, looks like I’m free after 10” blindSight

14. Pros – Easily learnable – Can convey lots of information Doesn’t always work – Requires direct interaction with device – Not usable in all scenarios (concerts, meetings, driving) auditory feedback

15. what about touch?

16. binary is easy

17. Vibratese Language - Communicate via vibrations Map alphabet and digits to 5 vibrators user with 65 hours training -> 90% accuracy, 38wpm System capable of delivering at 67wpm anything can be learned

18. vibratese language

19. human-to-human interaction

20. Tapping and Rubbing

21. Can we create tactile stimuli that feels like tapping or rubbing? Will people associate it with what they already associate with tapping and rubbing from human-human communication? research questions

22. Tapping Rubbing

23. soundTouch

24. what feels like a finger?

26. applications

27. in-car navigation

29. Task: Identify Given a tap rate its strength on a scale of 1-7. Given a tap rate its speed on a scale of 1- 7.

32. questionnaire How would you describe the tactile sensations you just experienced to someone who has not experienced them? Which aspects of the experience felt natural and which aspects did not? If your phone could generate these types of sensations, what would you like to use them for?

33. Tapping has human quality – 13 of 16 used the word ‘tap” in their description – 12 of 16 volunteered it had a human quality Fast and slow are perceptually different – 12 participants mentioned harder taps don’t feel natural – 5 said fast ones don’t feel natural Good for alerts – 6 volunteered that single taps would be good for silent environments – 7 volunteered they would be useful when vibrations can’t be felt (walking around) results

35. Taps have a number of characteristics that make them desirable – Quiet – Strong Rubbing is more subtle – Useful for in-the-hand scenarios Number of taps and rubs is key element – Sometimes, this has pre-learned meaning – Limits number of viable distinct icons design implications

36. mapping music to vibrations

37. PeopleTones: Buddy Proximity Notification

38. only two states, nearby and far away when a buddy is near, play their song if phone is in vibrate mode, vibrate the equivalent PeopleTones

39. measuring vibrations

41. generating vibrations

42. capturing the essence of music

43. high level approach just using beat doesn’t always work mapping lyrics doesn’t work well

44. remove noise isolate 6.6kHz to 17.6kHz components using 8 th order Butterworth Filter use amplitude threshold, to keep only components greater than the average

45. take running sum take running sum of absolute value, generate 1 value every 20ms this keeps length consistent

46. exaggerate features compose output from previous step with power function Ax n,x is sample, A and n are constants, 10<=A<15, 1<=n<=2

47. field study 3 groups of friends, 2 weeks

48. response to the cue

49. response to the cue’s information

50. lessons cues in the wild should be music higher comprehension rate when users select their own cues obtrusiveness of music cues was not a concern mapping music to vibration was most successful for people who knew the songs well semantic association is key

51. mapping physical objects to tactile feedback

52. problem tactile feedback is always the same, but visual and motor has directional aspect information is lost in the conversion

53. Add state information to tactile feedback – Hover state – Moved to the left/right/up/down Where do we put it? – Under the button – Make the sides move – Tapping? – solenoid solution

54. mapping speech to touch

55. research questions what are relevant characteristics of speech when mapping to tactile? how do users naturally perceive these to be mapped?

56. pilot study 5 common phrases from text messaging literature 20 vibration sequences Which phrase does this vibration feel like?

57. 5 phrases Hello. Goodbye. Where are you? Are you busy? I miss you.

59. lessons intonation is important syllables should match number of pulses duration should match (roughly)

60. potential applications learning to sign augmented sms messages messaging backchannel

61. thesis summary

62. touch mapping eyes free interaction humanobject hearing musicspeechsounds Tapping and Rubbing Simulating buttons People Tones People Tones Mapping language to tactile stimulus class sense example project People Tones People Tones lessons for application designers

63. timeline

64. Jan Apr Jun Sept Dec 2008 Ubicomp Doctoral colloquium Paper Deadlines Projects UIST Doctoral symposium CHI Doctoral consortium CHI Doctoral Symposiums Mapping language to tactile Simulating buttons

65. 2009 Paper Deadlines Projects Jan Apr Jun Sept Dec Mapping language to tactile Part 2: Field Deployment Mapping language to tactile Part 2: Field Deployment CHIUIST Simulating buttons Find a job

66. 2010 Paper Deadlines Projects Jan Apr Jun Sept Dec Dissertation Find a job Formalize music mapping

67. designing easily learnable eyes-free interaction Kevin Li University of California, San Diego

68. EXTRA SLIDES

69. Please listen carefully as our options have changed…

72. blindSight evaluation

73. interfaces Smartphone 2003 (sighted)BlindSight (eyes-free) vs.

74. task while “driving”idle (1) schedule appointments and (2) add contacts

75. 012345678 Was not missing information Knew position in the menu Knew what day/time I was at Felt in control of the conversation Better for setting meeting times Prefer if driving and talking Prefer Overall blindSightSmartphone Overall preference results

76. 1. brevity is good, but use in moderation clarification of navigation overrides brevity 2. predictable/modeless user interface is key 3. auditory feedback goes a long way even during phone call (disclaimer: need to study how it interferes with activities… driving) lessons