1. deWristified: Handwriting Inference Using Wrist-Based Motion Sensors Revisited
Raveen Wijewickrama
Anindya Maiti
Murtuza Jadliwala
University of Texas at San Antonio

2. Wrist Wearables
Extends the functionality of traditional wristwatches beyond timekeeping.
Captures rich contextual information about the wearer.
Enables several novel context-based applications.
8/19/2019
UTSA

3. Motion Sensors
Two main types of motion or inertial sensors:
Accelerometer: records device acceleration.
Gyroscope: records device angular rotation.
Accessing motion sensors on wearable devices:
All applications have access to motion sensors by default (also referred to as zero-permission sensors) on most wearable OSs.
Applications’ access to motion sensors cannot be regulated on most wearable OSs – we can’t turn them off!
Can an adversary take advantage of motion sensor data from a wrist-wearable device to infer private information inputted by the user’s device-wearing hand?
8/19/2019
UTSA

4. Inferring Private User Inputs (Using Wrist Wearables)
8/19/2019
UTSA

5. State-of-the-Art in Handwriting Recognition (Using Wrist Wearables)
Airwriting (Amma et al.)
Whiteboard writing (Arduser et al.)
Finger writing (Xu et al.)
Pen(cil) writing (Xia et al.)
8/19/2019
UTSA

6. Adversary Model Adversary has knowledge of the type of handwriting.
Adversary is able to record data from the target smartwatch’s accelerometer and gyroscope sensors.
Could employ a Trojan app for this!
Adversary’s Goal: To infer handwritten information using target user’s smartwatch sensors.
8/19/2019
UTSA

7. Limitations of Earlier Handwriting Recognition Studies (Using Wrist Wearables)
Airwriting (Amma et al.)
Finger writing (Xu et al.)
Custom-designed hand glove with very high precision sensors.
Use of Shimmer, a specialized sensing device intended for lab studies.
Our adversary relies on target user’s smartwatch or fitness band.
Only uppercase words.
Pen(cil) writing (Xia et al.)
Whiteboard writing (Arduser et al.)
Only lowercase alphabets.
Not generalized (training and testing data not from different participants).
Controlled data collection.
Only uppercase alphabets.
No handwriting activity detection.
8/19/2019
UTSA

8. Our Research How practical is handwriting inference when
Using consumer-grade wrist wearables,
Using generalized training and testing,
Writing in a uncontrolled and unconstrained manner, and
Both upper and lowercase alphabets are modeled ?
New Uncontrolled and Unconstrained Writing Data
Existing Models
8/19/2019
UTSA

9. Handwriting Inference Framework
8/19/2019
UTSA

10. Experimental Setup 28 participants for the four writing scenarios.
18 to 30 years of age
13 male, 15 female
Two different wrist-wearables.
Sony Smartwatch 3, LG Watch Urbane
Accelerometer and gyroscope recorded at 200Hz.
Participants provided with appropriate writing apparatus.
8/19/2019
UTSA

11. Writing Tasks (In-Lab)
Alphabets.
Individual alphabets one at a time.
Covered all 26 English alphabets in random order.
Each alphabet was written 10 times.
Both upper and lower cases.
Words.
4-8 alphabet words, from a vocabulary (Goldhahn et al. 2012).
Each participant wrote 20 words, in both upper and lower cases.
Sentence.
"the five boxing wizards jump quickly" in both upper and lower cases.
8/19/2019
UTSA

12. Writing Activity Recognition (Out of Lab)
2 participants.
Wore a smartwatch for an entire day.
Performed the four writing scenarios at random times.
Adversary’s Goal: To infer handwriting activity first, and then classify the handwritten text.
8/19/2019
UTSA

13. Replicated Inference Frameworks
Airwriting
Hidden Markov Model (HMM)
Whiteboard writing
Dynamic Time Warping (DTW)
Finger writing
Naive Bayes, Logistic Regression and Decision Tree classifiers
Pen(cil) writing
Random Forest classifier
8/19/2019
UTSA

14. Personalized Inference Accuracy
Writing Activity Detection:
56% recall and 57% precision for air and finger writing
39% recall and 47% precision for pencil writing
23% recall and 34% precision for whiteboard writing
8/19/2019
UTSA

15. Personalized Inference Accuracy (Whiteboard Writing)
Lowercase
Uppercase
8/19/2019
UTSA

16. Generalized Inference Accuracy
Writing Activity Detection:
35-40% recall for airwriting, whiteboard writing and pencil writing
Only 8% recall for finger writing
8/19/2019
UTSA

17. Factors Affecting Inference Accuracy
Number of Strokes.
8/19/2019
UTSA

18. Factors Affecting Inference Accuracy
Number of strokes for the same letter for different participants (lowercase).
Number of strokes for the same letter for different participants (uppercase).
8/19/2019
UTSA

19. Factors Affecting Inference Accuracy
Lowercase
Uppercase
Variance in number of strokes per alphabet per participant, averaged for all participants
8/19/2019
UTSA

20. Factors Affecting Inference Accuracy
Number of Strokes.
Order of Strokes.
Direction of Strokes.
8/19/2019
UTSA

21. Factors Affecting Inference Accuracy
8/19/2019
UTSA

22. Factors Affecting Inference Accuracy
Number of Strokes.
Order of Strokes.
Direction of Strokes.
Uppercase vs Lowercase.
Specialized Devices.
Airwriting (Amma et al.)
8/19/2019
UTSA

23. Conclusion
We investigated how wrist-wearable based handwriting inference attacks perform in realistic day-to-day writing situations.
Such inference attacks are unlikely to pose a substantial threat to users of current consume-grade smartwatches and fitness bands.
Primarily due to highly varying nature of handwriting.
Replicable artifacts:
8/19/2019
UTSA