1. Using Mobile Phones to Write in Air Sandip Agrawal, Ionut Constandache, Shravan Gaonkar, Kevin Cave, Frank DeRuyter, Romit Roy Choudhury

2. Wishful Thinking What if we could: Hold a phone like a pen, and write quick notes in the air.

3. More Specifically …  User holds phone like a pen  Writes short messages in air  Phone’s built-in accelerometer  Records acceleration due to hand motion  Converts signals  image  text  Displays text on screen  And sends a copy to user’s email address

4. Could be a Cool Toy But will anyone care?

5. Could be a Cool Toy But will anyone care?  Caption photos on-the-fly  Sketch maps, directions … even quick diagrams even quick diagrams  One handed use … while carrying baby/suitcase or while driving baby/suitcase or while driving Duke Univ.

6. Let’s Build It  Prototype on Nokia N95  Equipped with 3-axes accelerometer  Programmed usign Python + MATLAB  Did not expect this to be hard  But we had heavily under-estimated …

7. Design Challenges (1)  Background Vibration (Noise)  Sensitive accelerometers  Jitter from hand vibrations Approach: Smooth signal by filtering high freq. components Smooth signal by filtering high freq. components Suppress residual values < threshold Suppress residual values < threshold

8. Design Challenges (2)  Computing Displacement  Ambiguity when acceleration is zero  Can be constant velocity or stationary  Worse with noise Approach: Detect change of direction Detect change of direction Reset velocity between strokes Reset velocity between strokes Reset

9. Design Challenges (3)  Lifting Pen from the Paper  e.g., dotting the “i” and crossing the “t” Approach: Impulse on the Z axis during the lift Impulse on the Z axis during the lift Off the Record – On the Record Off the Record – On the Record P Z-axis spikes

10. Design Challenges (4)  Character transition  e.g., B vs 13 Approach: Use delimiters such as “.” (1.3 for 13) Use delimiters such as “.” (1.3 for 13) Soft leftward motion between alphabets Soft leftward motion between alphabets

11. Design Challenges (5)  Hands rotate while writing  Accelerometers only measure linear acceleration  Rotation injects ambiguity (Wii uses gyroscope) Approach: Hold phone in non-rotating grip Hold phone in non-rotating grip Determine angular orientation during the pause Determine angular orientation during the pause

12. Drawing a Rectangle Raw Accelerometer Reading

13. Noise Smoothing Residual Noise Suppression Velocity Plot after Avoiding Velocity Drifts Final Rectangle

14. Raw Accelerometer Data for M and Y

15. More PhonePen Words …

16. Character Recognition While geometric version can be displayed as image, text format necessary for searching English text expressed in strokes:

17. Simple Grammar Tree A B R

18. Incorporating Tolerance to Errors Standard “N” = Standard “N” = I + \ + I N s0 I \ s2 I I

19. Incorporating Tolerance to Errors Standard “N” = Standard “N” = I + \ + I N = I + + I However, imperfect human gestures induce distortions: N s0 I \ s2 I I DP BR \

20. Incorporating Tolerance to Errors Standard “N” = Standard “N” = I + \ + I N = I + + I However, imperfect human gestures induce distortions: N s0 I \ s2 I I DP BR \ I

21. Incorporating Tolerance to Errors Standard “N” = Standard “N” = I + \ + I N s0 I \ s2 I I DP BR \ I N = I + + I However, imperfect human gestures induce distortions:

22. Dense Grammar Tree Incorporating Tolerance to Errors Standard “N” = Standard “N” = I + \ + I N = I + + I N = I + \ + / N = / + \ + / N = / + I + / … However, imperfect human gestures induce distortions:

23. Spelling Correction

24. All things combined  PhonePoint Pen

25. On the Shoulders of Related Work  Stylus based sketch recognition  SketchREAD, Electronic Cocktail Napkin, Unistrokes  Gesture recognition  uWave, SmartQuil,  Vision based gesture recognition  TinyMotion, Microsoft Write in Air, VisionWand  Others …  Wiimote, Air-Mouse, SmartPen, NiiMe and others

26. Results

27.  10 test users (students)  6 trained (practiced each letter once, < 5min)  4 novices  Each user wrote 26 test characters in air Results  Metrics  Human readability  Character recognition  Word recognition (with spelling correction)  Writing speed

28. Readability and Recognition Human Readability Character Recognition

29. Per-User Accuracy … Speed

30. Short Words  Words chosen randomly from the dictionary  Written by trained users

31. Video Demo and Experience Search “PhonePen” in YouTube for full video

32. Thank You For demo video and more information, Visit synrg.ee.duke.edu

33. Swipe your credit card and sign in air … The vision is:

35. Of Course, Not a Product Yet  Lowercase character recognition  Cursive handwriting more complicated  Need smaller hand movements in air  Youtube comment: looking crazy waving hands  Slashdot: Nearby people can get injured  Quicker writing  3.02 seconds/alphabet is way too slow

36. Of Course, Not a Product Yet  What about longer messages  Sentences? Email replies?  Erasing alphabets  Shake your hands like using a eraser Ongoing work is pursuing these directions … Ongoing work is pursuing these directions …

37. Evaluation  10 test users (students)  6 trained (practiced each letter once, <5min)  4 novices  Each user wrote 26 test characters  5 patients from Duke Hospital  To analyze applicability into assistive technology

38. So, need a solution that is  Easy to use  Always-with-me  Allows sketching  Online/Searchable

39. Proposed Approach: PhonePoint Pen  User holds phone like a pen  Writes messages in air  Built-in accelerometer  Captures hand movement  Converts to image / text  Display text on screen  Also sent to user’s email address

40. Use Cases  Mashing with cameras  Superimposing tags on pictures  Sketching  Giving directions to your home  Quick diagram  One handed use  E.g., carrying a suitcase Duke University

41. Use Cases  Applications in healthcare  Hand movement disabilities  Speech problems after surgery  Military and emergency response  Quick and silent updates

42. Thank You! email: romit@ee.duke.eduromit@ee.duke.edu Web: http://synrg.ee.duke.edu/http://synrg.ee.duke.edu/

43. More PhonePen words … 20 words tested with online OCR web service. 78% accuracy 20 words tested with online OCR web service. 78% accuracy

44. Noting small pieces of information, quickly and effortlessly can be useful So, what are the options... Call John @ 6658 Buy Milk Right on 3rd traffic light

45. Velocity Plot after Avoiding Velocity Drifts Final Rectangle

46. Noting small pieces of information, quickly and effortlessly can be useful But aren’t there several solutions... Buy Milk

47. State of the Art  Sticky notes … Difficult to organize electronically  Difficult to organize electronically  Not always handy  Typing on phone keyboards Tiny keyboard sizes  Tiny keyboard sizes  Small inter-key spacing  Audio recording Cannot sketch diagrams  Cannot sketch diagrams  Time consuming to browse through voice messages

48. So, need a solution that is  Easy to use  Always-with-me  Allows sketching  Online/Searchable

49. Incorporating Tolerance to Errors

50. Patient Results

51. Incorporating Tolerance to Errors Standard “N” = Standard “N” = I + \ + I N = I + + I N = I + \ + / N = / + \ + / N = / + I + / … However, human gestures induce distortions: N

52. Incorporating Tolerance to Errors Standard “N” = Standard “N” = I + \ + I N = I + + I N = I + \ + / N = / + \ + / N = / + I + / … However, human gestures induce distortions: N s0 I \ s2 I I DP BR \

53. Incorporating Tolerance to Errors Standard “N” = Standard “N” = I + \ + I N = I + + I N = I + \ + / N = / + \ + / N = / + I + / … However, human gestures induce distortions: N s0 I \ s2 I I DP BR \

54. Incorporating Tolerance to Errors Standard “N” = Standard “N” = I + \ + I N = I + + I N = I + \ + / N = / + \ + / N = / + I + / … However, human gestures induce distortions: N s0 I \ s2 I I DP BR \ I

55. Incorporating Tolerance to Errors Standard “N” = Standard “N” = I + \ + I N = I + + I N = I + \ + / N = / + \ + / N = / + I + / … However, human gestures induce distortions: N s0 I \ s2 I I DP BR \ I

56. Readability and Recognition