designing easily learnable eyes-free interaction Kevin Li University of California, San Diego

can we make easily learnable eyes-free interfaces by leveraging existing learned associations?

touch mapping eyes free interaction humanobject hearing musicspeechsounds Tapping and Rubbing Simulating buttons People Tones People Tones Mapping language to tactile stimulus class sense example project People Tones People Tones lessons for application designers

why eyes free?

PCs… PC screens have the users’ undivided attention  design for the visual channel

environment

screen-less device

can’t see screen

visual impairment

social factors

easily learnable?

binary is easy

Vibratese Language - Communicate via vibrations Map alphabet and digits to 5 vibrators user with 65 hours training -> 90% accuracy, 38wpm System capable of delivering at 67wpm anything can be learned

vibratese language

Ok, so let’s just translate all text from visual to auditory – Easily learnable – High bandwidth Wait, that sounds familiar auditory feedback

Please listen carefully as our options have changed…

blindSight evaluation

interfaces Smartphone 2003 (sighted)BlindSight (eyes-free) vs.

task while “driving”idle (1) schedule appointments and (2) add contacts

Was not missing information Knew position in the menu Knew what day/time I was at Felt in control of the conversation Better for setting meeting times Prefer if driving and talking Prefer Overall blindSightSmartphone Overall preference results

1. brevity is good, but use in moderation clarification of navigation overrides brevity 2. predictable/modeless user interface is key 3. auditory feedback goes a long way even during phone call (disclaimer: need to study how it interferes with activities… driving) lessons

Requires direct interaction with device Not usable in all scenarios – Loud environments (concerts) – Social environments – Driving and talking on the phone auditory feedback doesn’t always work

what about touch?

replace visual applications…

… new applications?

human-to-human interaction

Tapping and Rubbing

Can we create tactile stimuli that feels like tapping or rubbing? Will people associate it with what they already associate with tapping and rubbing in human- human communication? research questions

Tapping Rubbing

soundTouch

what feels like a finger?

applications

in-car navigation

Task 1: Distinguish Given two taps of different strength, which one feels stronger? Given two taps of different frequency, which one feels faster?

error % for distinguish amplitude Level of first stimulus Level of second stimulus

error % for distinguish frequency Level of first stimulus Level of second stimulus ConstantDuration ConstantNumber

Task 2: Identify Given a tap rate its strength on a scale of 1-7. Given a tap rate its speedon a scale of 1- 7.

questionnaire How would you describe the tactile sensations you just experienced to someone who has not experienced them? Which aspects of the experience felt natural and which aspects did not? If your phone could generate these types of sensations, what would you like to use them for?

Tapping has human quality – 13 of 16 used the word ‘tap” in their description – 12 of 16 volunteered it had a human quality Fast and slow are perceptually different – 12 participants mentioned harder taps don’t feel natural – 5 said fast ones don’t feel natural Good for alerts – 6 volunteered that single taps would be good for silent environments – 7 volunteered they would be useful when vibrations can’t be felt (walking around) results

Taps have a number of characteristics that make them desirable – Quiet – Strong Rubbing is more subtle – Useful for in-the-hand scenarios Number of taps and rubs is key element – Sometimes, this has pre-learned meaning – Limits number of viable distinct icons design implications

mapping music to vibrations

PeopleTones: Buddy Proximity Notification

only two states, nearby and far away when a buddy is near, play their song if phone is in vibrate mode, vibrate the equivalent PeopleTones

measuring vibrations

generating vibrations

capturing essence of music just using beat doesn’t always work mapping lyrics doesn’t work well

remove noise isolate 6.6kHz to 17.6kHz components using 8 th order Butterworth Filter use amplitude threshold, to keep only components greater than the average

take running sum take running sum of absolute value, generate 1 value every 20ms this keeps length consistent

exaggerate features compose output from previous step with power function Ax n,x is sample, A and n are constants, 10<=A<15, 1<=n<=2

field study 3 groups of friends, 2 weeks

response to the cue

response to the cue’s information

lessons cues in the wild should be music higher comprehension rate when users select their own cues obtrusiveness of music cues was not a concern mapping music to vibration was most successful for people who knew the songs well semantic association is key

mapping physical objects to tactile feedback

problem tactile feedback is always the same, but visual and motor has directional aspect information is lost in the conversion

Add state information to tactile feedback – Hover state – Moved to the left/right/up/down Where do we put it? – Under the button – Make the sides move – Tapping? – solenoid solution

mapping speech to touch

research questions what are relevant characteristics of speech when mapping to tactile? how do users naturally perceive these to be mapped?

pilot study 5 common phrases from text messaging literature 20 vibration sequences Which phrase does this vibration feel like?

5 phrases Hello. Goodbye. Where are you? Are you busy? I miss you.

lessons intonation is important syllables should match number of pulses duration should match (roughly)

potential applications learning to sign augmented sms messages messaging backchannel

thesis summary

touch mapping eyes free interaction humanobject hearing musicspeechsounds Tapping and Rubbing Simulating buttons People Tones People Tones Mapping language to tactile stimulus class sense example project People Tones People Tones lessons for application designers

timeline

Jan Apr Jun Sept Dec 2008 Ubicomp Doctoral colloquium Paper Deadlines Projects UIST Doctoral symposium CHI Doctoral consortium CHI Doctoral Symposiums Mapping language to tactile Simulating buttons

2009 Paper Deadlines Projects Jan Apr Jun Sept Dec Mapping language to tactile Part 2: Field Deployment Mapping language to tactile Part 2: Field Deployment CHIUIST Simulating buttons Find a job

2010 Paper Deadlines Projects Jan Apr Jun Sept Dec Dissertation Find a job Formalize music mapping

designing easily learnable eyes-free interaction Kevin Li University of California, San Diego