1. The Sound of One Hand: A Wrist-mounted Bio-acoustic Fingertip Gesture Interface Brian Amento, Will Hill AT&T Labs – Research Loren Terveen University of Minnesota

2. Outline Motivation Gesture Interfaces Signal Classifiers Prototype Applications Future Work

3. Motivation Small wearable digital devices increasingly popular (Cellphones, PDAs, Rios, etc..) Nonlinear access to linear media will increase –Voicemail, Music, Video, Radio, Text –Controls: Device Select, Play, Stop, Scan forward, Scan backward, Faster, Slower, Item Select, Exit

4. Current Interfaces to Mobile Devices Two-handed control mechanisms Pressing device buttons Writing/selecting with stylus –Un-holstering a wearable is a pain (i.e., wristwatches beat pocket watches) Speech recognition –Noise or social setting may rule out voice control Our Goal: Invisible, weightless, un-tethered and cost-free

5. How about a gesture interface?

6. Body tracking Teresa Martin 1997 Polhemus 2000

7. Datagloves

8. Image hand tracking Cullen Jennings, 1999

9. Our Approach Natural fingertip gestures

10. Whats natural Small - max displacement of 5 cm Gentle, < 10% of pressing strength (e.g. no finger snap) Few gestures, little memory work Avoid ring and pinky finger Examples: –Thumb as anvil - index, middle as hammer –Thumbpad to fingerpad –Thumbpad to fingernail edge

11. Fingertip Gestures Tap, double tap Finger and thumb pads rub Money gesture and reverse Finger and thumb pads press Soft Flick

12. Fingertip Gesture Interface Wristband-mounted piezo-electric contact microphones positioned on the styloid bones Sense bone conducted sounds produced by gentle fingertip gestures

13. Simple Classifier Allows real-time analysis and control 800 samples every 10 th of a second Take max absolute, quantize to 10 levels Finite state machine outputs Taps and Rubs –Intermediate states filter background noise –Buffer states allow continuous gestures Surprisingly accurate: ~90%

14. Example Signals

15. More Sophisticated Classifier Noticeable differences in audio signals Hidden Markov Models Gesture and noise models trained with sampled data Confidence levels for each trained gesture

16. HMM Classifier Accuracy Using 3 subjects, collected 100 instances of gestures rub, tap and flick 80 used for training, 20 for testing Accuracy Tap55/60 (92%) Rub59/60 (98%) Flick56/60 (96%)

17. Wrist Display Prototype Timex Internet Messenger watch Tap to cycle through messages Double-tap to rewind

18. Other Prototypes Cellphone dialing application –Rub scrolls list in one direction –Tap dials phone number Powerpoint slide control –Tap moves forward one slide –Double tap moves back

19. Future Work Miniaturization of device –Hitachi SH5 controller Improved gesture classifiers Finger Identification –Analyze signals from multiple microphone locations User Studies –Usefulness: Compare performance to current cellphone, PDA and desktop control interfaces. –Social impact: Study how users exploit private control techniques to mobile devices

20. Conclusion Fingertip gestures –sensed acoustically at the wrist –can be communicated wirelessly to nearby devices –show promise as a control method.