1. Using Mobile Phones to Write in Air
PhonePoint Pen:
Using Mobile Phones to Write in Air
Romit Roy Choudhury Assistant Professor

2. Noting small pieces of information, quickly and effortlessly
can be useful
So, what are the options ...
Buy Milk

3. State of the Art Sticky notes … Typing on keyboard Audio recording
organization is a nightmare
not handy
Typing on keyboard
tiny keyboard sizes
small inter-key spacing
Audio recording
cannot sketch information
time consuming to browse through voice messages

4. So, need a solution that is
Easy to use
Always-with-me
Allows sketching
Searchable

5. Proposed Approach: PhonePoint Pen
User can write messages in air
holding the phone like a pen
Use built-in accelerometer
capture hand movement
Decode text and image
sent to user’s address for future use

6. Use Cases Mashing with cameras Sketching One handed use
Duke University
Use Cases
Mashing with cameras
superimposing text on pictures
Sketching
exchanging figures between caller/callee
One handed use
e.g., carrying a suitcase

7. Prototype
Current prototype implemented on Nokia N95 mobile phones
several design challenges emerge …

8. Design Challenges (1) Lack of a Gyroscope Proposed Approach:
Accelerometers only measure linear acceleration
Linear Movements – Rotation Ambiguity
Proposed Approach:
Hold Phone in Non-rotating Grip
Determine Angular Orientation during the pause P

9. Background Vibration (Noise)
Design Challenges (2)
Background Vibration (Noise)
Sensitive Accelerometers
Significant Jitter by hand vibrations
Proposed Approach:
Smooth the accelerometer readings with moving average over last n (=7) readings
Suppress acceleration values < 0.5 P

10. Computing Displacement
Design Challenges (3)
Computing Displacement
Erroneous Acceleration Reading
Ambiguity when acceleration is zero
Proposed Approach:
Detect Pause Using Moving Window
Reset Velocity to Zero in Between Strokes P

11. Lifting Pen from the Paper
Design Challenges (4)
Lifting Pen from the Paper
e.g., dotting the “i” and crossing the “t”
Proposed Approach:
Impulse on the Z axis during the lift
Off the Record – On the Record P

12. Raw Accelerometer Reading
Drawing a Rectangle
Raw Accelerometer Reading

13. Noise Smoothing using Moving Average
Background Noise Suppression

14. Velocity Plot after Avoiding Velocity Drifts
Final Rectangle

15. Raw Accelerometer Data for M and i

16. More PhonePen words …

17. Ongoing Work
Smaller hand movements in air
Lowercase character recognition
Increased robustness and reliability

18. Swipe your credit card and sign in air …
Our vision:
Swipe your credit card and sign in air …

19. Web: http://synrg.ee.duke.edu/
Thank You!
Web: