1. designing easily learnable eyes-free interaction Kevin Li University of California, San Diego

2. can we make easily learnable eyes-free interfaces by leveraging existing learned associations?

3. touch mapping eyes free interaction humanobject hearing musicspeechsounds Tapping and Rubbing Simulating buttons People Tones People Tones Mapping language to tactile stimulus class sense example project People Tones People Tones lessons for application designers

4. why eyes free?

5. PCs… PC screens have the users’ undivided attention  design for the visual channel

6. environment

7. screen-less device

8. can’t see screen

9. visual impairment

10. social factors

11. easily learnable?

12. binary is easy

13. Vibratese Language - Communicate via vibrations Map alphabet and digits to 5 vibrators user with 65 hours training -> 90% accuracy, 38wpm System capable of delivering at 67wpm anything can be learned

14. vibratese language

15. Ok, so let’s just translate all text from visual to auditory – Easily learnable – High bandwidth Wait, that sounds familiar auditory feedback

16. Please listen carefully as our options have changed…

19. blindSight evaluation

20. interfaces Smartphone 2003 (sighted)BlindSight (eyes-free) vs.

21. task while “driving”idle (1) schedule appointments and (2) add contacts

22. 012345678 Was not missing information Knew position in the menu Knew what day/time I was at Felt in control of the conversation Better for setting meeting times Prefer if driving and talking Prefer Overall blindSightSmartphone Overall preference results

23. 1. brevity is good, but use in moderation clarification of navigation overrides brevity 2. predictable/modeless user interface is key 3. auditory feedback goes a long way even during phone call (disclaimer: need to study how it interferes with activities… driving) lessons

24. Requires direct interaction with device Not usable in all scenarios – Loud environments (concerts) – Social environments – Driving and talking on the phone auditory feedback doesn’t always work

25. what about touch?

26. replace visual applications…

27. … new applications?

28. human-to-human interaction

29. Tapping and Rubbing

30. Can we create tactile stimuli that feels like tapping or rubbing? Will people associate it with what they already associate with tapping and rubbing in human- human communication? research questions

31. Tapping Rubbing

32. soundTouch

33. what feels like a finger?

35. applications

36. in-car navigation

37. Task 1: Distinguish Given two taps of different strength, which one feels stronger? Given two taps of different frequency, which one feels faster?

38. error % for distinguish amplitude Level of first stimulus Level of second stimulus

39. error % for distinguish frequency Level of first stimulus Level of second stimulus ConstantDuration ConstantNumber

40. Task 2: Identify Given a tap rate its strength on a scale of 1-7. Given a tap rate its speedon a scale of 1- 7.

43. questionnaire How would you describe the tactile sensations you just experienced to someone who has not experienced them? Which aspects of the experience felt natural and which aspects did not? If your phone could generate these types of sensations, what would you like to use them for?

44. Tapping has human quality – 13 of 16 used the word ‘tap” in their description – 12 of 16 volunteered it had a human quality Fast and slow are perceptually different – 12 participants mentioned harder taps don’t feel natural – 5 said fast ones don’t feel natural Good for alerts – 6 volunteered that single taps would be good for silent environments – 7 volunteered they would be useful when vibrations can’t be felt (walking around) results

45. Taps have a number of characteristics that make them desirable – Quiet – Strong Rubbing is more subtle – Useful for in-the-hand scenarios Number of taps and rubs is key element – Sometimes, this has pre-learned meaning – Limits number of viable distinct icons design implications

46. mapping music to vibrations

47. PeopleTones: Buddy Proximity Notification

48. only two states, nearby and far away when a buddy is near, play their song if phone is in vibrate mode, vibrate the equivalent PeopleTones

49. measuring vibrations

51. generating vibrations

52. capturing essence of music just using beat doesn’t always work mapping lyrics doesn’t work well

53. remove noise isolate 6.6kHz to 17.6kHz components using 8 th order Butterworth Filter use amplitude threshold, to keep only components greater than the average

54. take running sum take running sum of absolute value, generate 1 value every 20ms this keeps length consistent

55. exaggerate features compose output from previous step with power function Ax n,x is sample, A and n are constants, 10<=A<15, 1<=n<=2

56. field study 3 groups of friends, 2 weeks

57. response to the cue

58. response to the cue’s information

59. lessons cues in the wild should be music higher comprehension rate when users select their own cues obtrusiveness of music cues was not a concern mapping music to vibration was most successful for people who knew the songs well semantic association is key

60. mapping physical objects to tactile feedback

61. problem tactile feedback is always the same, but visual and motor has directional aspect information is lost in the conversion

62. Add state information to tactile feedback – Hover state – Moved to the left/right/up/down Where do we put it? – Under the button – Make the sides move – Tapping? – solenoid solution

63. mapping speech to touch

64. research questions what are relevant characteristics of speech when mapping to tactile? how do users naturally perceive these to be mapped?

65. pilot study 5 common phrases from text messaging literature 20 vibration sequences Which phrase does this vibration feel like?

66. 5 phrases Hello. Goodbye. Where are you? Are you busy? I miss you.

68. lessons intonation is important syllables should match number of pulses duration should match (roughly)

69. potential applications learning to sign augmented sms messages messaging backchannel

70. thesis summary

71. touch mapping eyes free interaction humanobject hearing musicspeechsounds Tapping and Rubbing Simulating buttons People Tones People Tones Mapping language to tactile stimulus class sense example project People Tones People Tones lessons for application designers

72. timeline

73. Jan Apr Jun Sept Dec 2008 Ubicomp Doctoral colloquium Paper Deadlines Projects UIST Doctoral symposium CHI Doctoral consortium CHI Doctoral Symposiums Mapping language to tactile Simulating buttons

74. 2009 Paper Deadlines Projects Jan Apr Jun Sept Dec Mapping language to tactile Part 2: Field Deployment Mapping language to tactile Part 2: Field Deployment CHIUIST Simulating buttons Find a job

75. 2010 Paper Deadlines Projects Jan Apr Jun Sept Dec Dissertation Find a job Formalize music mapping

76. designing easily learnable eyes-free interaction Kevin Li University of California, San Diego