PhonePoint Pen: Using Mobile Phones to Write in Air Sandip Agrawal, Ionut Constandache, Shravan Gaonkar, Romit Roy Choudhury ACM MobiHeld 2009

Noting small pieces of information, quickly and effortlessly can be useful But aren’t there several solutions... Buy Milk

State of the Art  Sticky notes … Difficult to organize electronically  Difficult to organize electronically  Not always handy  Typing on phone keyboards Tiny keyboard sizes  Tiny keyboard sizes  Small inter-key spacing  Audio recording Cannot sketch diagrams  Cannot sketch diagrams  Time consuming to browse through voice messages

So, need a solution that is  Easy to use  Always-with-me  Allows sketching  Online/Searchable

Proposed Approach: PhonePoint Pen  User holds phone like a pen  Writes messages in air  Built-in accelerometer  Captures hand movement  Converts to image / text  Display text on screen  Also sent to user’s address

Use Cases  Mashing with cameras  Superimposing text on pictures  Sketching  Giving directions to your home  Exchanging figures on the fly  One handed use  E.g., carrying a suitcase Duke University

Use Cases  Applications in healthcare  Hand movement disabilities  Speech problems after surgery  Military and emergency response  Quick and silent updates

Prototype Current prototype on Nokia N95 Equipped with 3-axes accelerometer Programmed usign Python + MATLAB several design challenges emerge …

Design Challenges (1)  Hands rotate while writing  Accelerometers only measure linear acceleration  Rotation injects ambiguity (Wii uses gyroscope) Proposed Approach: Hold phone in non-rotating grip Hold phone in non-rotating grip Determine angular orientation during the pause Determine angular orientation during the pause

Design Challenges (2)  Background Vibration (Noise)  Sensitive accelerometers  Jitter from hand vibrations Proposed Approach: Smooth the accelerometer readings with moving average Smooth the accelerometer readings with moving average Suppress acceleration values < threshold Suppress acceleration values < threshold

Design Challenges (3)  Computing Displacement  Ambiguity when acceleration is zero  Can be constant velocity or stationary  Worse with noise Proposed Approach: Detect change of direction Detect change of direction Reset velocity between strokes Reset velocity between strokes Reset

Design Challenges (4)  Lifting Pen from the Paper  e.g., dotting the “i” and crossing the “t” Proposed Approach: Impulse on the Z axis during the lift Impulse on the Z axis during the lift Off the Record – On the Record Off the Record – On the Record P Z-axis spikes

Design Challenges (5)  Character transition  e.g., B vs 13 Proposed Approach: Use delimited such as “.” Use delimited such as “.” 1.3 for for 13

Drawing a Rectangle Raw Accelerometer Reading Raw Accelerometer Reading

Noise Smoothing using Moving Average Background Noise Suppression Velocity Plot after Avoiding Velocity Drifts Final Rectangle

Raw Accelerometer Data for M and i

Character Recognition

More PhonePen words …

20 words tested with online OCR web service. 78% accuracy 20 words tested with online OCR web service. 78% accuracy

Results

Video Demo

Of Course, Not a Product Yet  Lowercase character recognition  Cursive handwriting more complicated  Need smaller hand movements in air  Youtube comment: looking crazy waving hands  Slashdot: Nearby people can get injured  Need increased robustness  20% failure when writing in air  5% failure when writing on a surface

Of Course, Not a Product Yet  What about longer messages  Sentences? replies?  Erasing alphabets  Shake your hands like using a eraser Ongoing work is pursuing these directions … Ongoing work is pursuing these directions …

Swipe your credit card and sign in air … The vision is:

Thanks Visit Systems Networking Research Group Duke University Google “synrg duke”

Thank You! Web:

Noting small pieces of information, quickly and effortlessly can be useful So, what are the options... Call 6658 Buy Milk Right on 3rd traffic light

Velocity Plot after Avoiding Velocity Drifts Final Rectangle