The Sound of One Hand: A Wrist-mounted Bio-acoustic Fingertip Gesture Interface Brian Amento, Will Hill AT&T Labs – Research Loren Terveen University of Minnesota

Outline Motivation Gesture Interfaces Signal Classifiers Prototype Applications Future Work

Motivation Small wearable digital devices increasingly popular (Cellphones, PDAs, Rios, etc..) Nonlinear access to linear media will increase –Voic , Music, Video, Radio, Text –Controls: Device Select, Play, Stop, Scan forward, Scan backward, Faster, Slower, Item Select, Exit

Current Interfaces to Mobile Devices Two-handed control mechanisms Pressing device buttons Writing/selecting with stylus –Un-holstering a wearable is a pain (i.e., wristwatches beat pocket watches) Speech recognition –Noise or social setting may rule out voice control Our Goal: Invisible, weightless, un-tethered and cost-free

How about a gesture interface?

Body tracking Teresa Martin 1997 Polhemus 2000

Datagloves

Image hand tracking Cullen Jennings, 1999

Our Approach Natural fingertip gestures

Whats natural Small - max displacement of 5 cm Gentle, < 10% of pressing strength (e.g. no finger snap) Few gestures, little memory work Avoid ring and pinky finger Examples: –Thumb as anvil - index, middle as hammer –Thumbpad to fingerpad –Thumbpad to fingernail edge

Fingertip Gestures Tap, double tap Finger and thumb pads rub Money gesture and reverse Finger and thumb pads press Soft Flick

Fingertip Gesture Interface Wristband-mounted piezo-electric contact microphones positioned on the styloid bones Sense bone conducted sounds produced by gentle fingertip gestures

Simple Classifier Allows real-time analysis and control 800 samples every 10 th of a second Take max absolute, quantize to 10 levels Finite state machine outputs Taps and Rubs –Intermediate states filter background noise –Buffer states allow continuous gestures Surprisingly accurate: ~90%

Example Signals

More Sophisticated Classifier Noticeable differences in audio signals Hidden Markov Models Gesture and noise models trained with sampled data Confidence levels for each trained gesture

HMM Classifier Accuracy Using 3 subjects, collected 100 instances of gestures rub, tap and flick 80 used for training, 20 for testing Accuracy Tap55/60 (92%) Rub59/60 (98%) Flick56/60 (96%)

Wrist Display Prototype Timex Internet Messenger watch Tap to cycle through messages Double-tap to rewind

Other Prototypes Cellphone dialing application –Rub scrolls list in one direction –Tap dials phone number Powerpoint slide control –Tap moves forward one slide –Double tap moves back

Future Work Miniaturization of device –Hitachi SH5 controller Improved gesture classifiers Finger Identification –Analyze signals from multiple microphone locations User Studies –Usefulness: Compare performance to current cellphone, PDA and desktop control interfaces. –Social impact: Study how users exploit private control techniques to mobile devices

Conclusion Fingertip gestures –sensed acoustically at the wrist –can be communicated wirelessly to nearby devices –show promise as a control method.