Using Mobile Phones to Write in Air Sandip Agrawal, Ionut Constandache, Shravan Gaonkar, Kevin Cave, Frank DeRuyter, Romit Roy Choudhury

Wishful Thinking What if we could: Hold a phone like a pen, and write quick notes in the air.

More Specifically …  User holds phone like a pen  Writes short messages in air  Phone’s built-in accelerometer  Records acceleration due to hand motion  Converts signals  image  text  Displays text on screen  And sends a copy to user’s address

Could be a Cool Toy But will anyone care?

Could be a Cool Toy But will anyone care?  Caption photos on-the-fly  Sketch maps, directions … even quick diagrams even quick diagrams  One handed use … while carrying baby/suitcase or while driving baby/suitcase or while driving Duke Univ.

Let’s Build It  Prototype on Nokia N95  Equipped with 3-axes accelerometer  Programmed usign Python + MATLAB  Did not expect this to be hard  But we had heavily under-estimated …

Design Challenges (1)  Background Vibration (Noise)  Sensitive accelerometers  Jitter from hand vibrations Approach: Smooth signal by filtering high freq. components Smooth signal by filtering high freq. components Suppress residual values < threshold Suppress residual values < threshold

Design Challenges (2)  Computing Displacement  Ambiguity when acceleration is zero  Can be constant velocity or stationary  Worse with noise Approach: Detect change of direction Detect change of direction Reset velocity between strokes Reset velocity between strokes Reset

Design Challenges (3)  Lifting Pen from the Paper  e.g., dotting the “i” and crossing the “t” Approach: Impulse on the Z axis during the lift Impulse on the Z axis during the lift Off the Record – On the Record Off the Record – On the Record P Z-axis spikes

Design Challenges (4)  Character transition  e.g., B vs 13 Approach: Use delimiters such as “.” (1.3 for 13) Use delimiters such as “.” (1.3 for 13) Soft leftward motion between alphabets Soft leftward motion between alphabets

Design Challenges (5)  Hands rotate while writing  Accelerometers only measure linear acceleration  Rotation injects ambiguity (Wii uses gyroscope) Approach: Hold phone in non-rotating grip Hold phone in non-rotating grip Determine angular orientation during the pause Determine angular orientation during the pause

Drawing a Rectangle Raw Accelerometer Reading

Noise Smoothing Residual Noise Suppression Velocity Plot after Avoiding Velocity Drifts Final Rectangle

Raw Accelerometer Data for M and Y

More PhonePen Words …

Character Recognition While geometric version can be displayed as image, text format necessary for searching English text expressed in strokes:

Simple Grammar Tree A B R

Incorporating Tolerance to Errors Standard “N” = Standard “N” = I + \ + I N s0 I \ s2 I I

Incorporating Tolerance to Errors Standard “N” = Standard “N” = I + \ + I N = I + + I However, imperfect human gestures induce distortions: N s0 I \ s2 I I DP BR \

Incorporating Tolerance to Errors Standard “N” = Standard “N” = I + \ + I N = I + + I However, imperfect human gestures induce distortions: N s0 I \ s2 I I DP BR \ I

Incorporating Tolerance to Errors Standard “N” = Standard “N” = I + \ + I N s0 I \ s2 I I DP BR \ I N = I + + I However, imperfect human gestures induce distortions:

Dense Grammar Tree Incorporating Tolerance to Errors Standard “N” = Standard “N” = I + \ + I N = I + + I N = I + \ + / N = / + \ + / N = / + I + / … However, imperfect human gestures induce distortions:

Spelling Correction

All things combined  PhonePoint Pen

On the Shoulders of Related Work  Stylus based sketch recognition  SketchREAD, Electronic Cocktail Napkin, Unistrokes  Gesture recognition  uWave, SmartQuil,  Vision based gesture recognition  TinyMotion, Microsoft Write in Air, VisionWand  Others …  Wiimote, Air-Mouse, SmartPen, NiiMe and others

Results

 10 test users (students)  6 trained (practiced each letter once, < 5min)  4 novices  Each user wrote 26 test characters in air Results  Metrics  Human readability  Character recognition  Word recognition (with spelling correction)  Writing speed

Readability and Recognition Human Readability Character Recognition

Per-User Accuracy … Speed

Short Words  Words chosen randomly from the dictionary  Written by trained users

Video Demo and Experience Search “PhonePen” in YouTube for full video

Thank You For demo video and more information, Visit synrg.ee.duke.edu

Swipe your credit card and sign in air … The vision is:

Of Course, Not a Product Yet  Lowercase character recognition  Cursive handwriting more complicated  Need smaller hand movements in air  Youtube comment: looking crazy waving hands  Slashdot: Nearby people can get injured  Quicker writing  3.02 seconds/alphabet is way too slow

Of Course, Not a Product Yet  What about longer messages  Sentences? replies?  Erasing alphabets  Shake your hands like using a eraser Ongoing work is pursuing these directions … Ongoing work is pursuing these directions …

Evaluation  10 test users (students)  6 trained (practiced each letter once, <5min)  4 novices  Each user wrote 26 test characters  5 patients from Duke Hospital  To analyze applicability into assistive technology

So, need a solution that is  Easy to use  Always-with-me  Allows sketching  Online/Searchable

Proposed Approach: PhonePoint Pen  User holds phone like a pen  Writes messages in air  Built-in accelerometer  Captures hand movement  Converts to image / text  Display text on screen  Also sent to user’s address

Use Cases  Mashing with cameras  Superimposing tags on pictures  Sketching  Giving directions to your home  Quick diagram  One handed use  E.g., carrying a suitcase Duke University

Use Cases  Applications in healthcare  Hand movement disabilities  Speech problems after surgery  Military and emergency response  Quick and silent updates

Thank You! Web:

More PhonePen words … 20 words tested with online OCR web service. 78% accuracy 20 words tested with online OCR web service. 78% accuracy

Noting small pieces of information, quickly and effortlessly can be useful So, what are the options... Call 6658 Buy Milk Right on 3rd traffic light

Velocity Plot after Avoiding Velocity Drifts Final Rectangle

Noting small pieces of information, quickly and effortlessly can be useful But aren’t there several solutions... Buy Milk

State of the Art  Sticky notes … Difficult to organize electronically  Difficult to organize electronically  Not always handy  Typing on phone keyboards Tiny keyboard sizes  Tiny keyboard sizes  Small inter-key spacing  Audio recording Cannot sketch diagrams  Cannot sketch diagrams  Time consuming to browse through voice messages

So, need a solution that is  Easy to use  Always-with-me  Allows sketching  Online/Searchable

Incorporating Tolerance to Errors

Patient Results

Incorporating Tolerance to Errors Standard “N” = Standard “N” = I + \ + I N = I + + I N = I + \ + / N = / + \ + / N = / + I + / … However, human gestures induce distortions: N

Incorporating Tolerance to Errors Standard “N” = Standard “N” = I + \ + I N = I + + I N = I + \ + / N = / + \ + / N = / + I + / … However, human gestures induce distortions: N s0 I \ s2 I I DP BR \

Incorporating Tolerance to Errors Standard “N” = Standard “N” = I + \ + I N = I + + I N = I + \ + / N = / + \ + / N = / + I + / … However, human gestures induce distortions: N s0 I \ s2 I I DP BR \

Incorporating Tolerance to Errors Standard “N” = Standard “N” = I + \ + I N = I + + I N = I + \ + / N = / + \ + / N = / + I + / … However, human gestures induce distortions: N s0 I \ s2 I I DP BR \ I

Incorporating Tolerance to Errors Standard “N” = Standard “N” = I + \ + I N = I + + I N = I + \ + / N = / + \ + / N = / + I + / … However, human gestures induce distortions: N s0 I \ s2 I I DP BR \ I

Readability and Recognition