AsiaCCS 2016 Smartwatch-Based Keystroke Inference Attacks and Context-Aware Protection Mechanisms Anindya Maiti, Oscar Armbruster, Murtuza Jadliwala, Jibo He {axmaiti, oxarmbruster, murtuza.jadliwala, jibo.he}@wichita.edu

Keystrokes and Privacy We type more than ever before. It is important to protect from eavesdropping, because often the typed information is sensitive. Typing has become an integral part of our lives. But there’s always someone watching! Friday, February 22, 2019

Keystrokes and Privacy Credit Card Information Friday, February 22, 2019

Keystrokes and Privacy Tax Filing Friday, February 22, 2019

Keystrokes and Privacy Emails/Messages And so on… Friday, February 22, 2019

Wi-Fi Channel State Information Side-Channel Attacks Channel Type Common Limitation Berger et al. [CCS’06] Acoustic Change in position of either keyboard or eavesdropping device renders previous training data useless! Marquardt et al. [CCS’11] Surface Vibration Ali et al. [MobiCom’15] Wi-Fi Channel State Information For this talk let’s focus on eavesdropping attacks similar to our work, also know as side-channel attacks. They are called side-channel because the primary purpose of the eavesdropping device/sensor was not intended for eavesdropping. Friday, February 22, 2019

Day 1 The fundamental principle behind such attacks is that each key produces unique characteristics, for example, due to the distance from the eavesdropping device. Here, we look at Marquardt et al.’s setup using smartphone motion sensors to capture surface vibration. Keys near… Keys far…will produce different characteristics in surface vibration. Day 2 Friday, February 22, 2019

Smartwatches Wristwatch with functionality well beyond timekeeping. Miniaturized computer. Friday, February 22, 2019

Behind the Scenes Sensors Motion Microphone GPS Camera Ambient Light Temperature … Sensors such as GPS and camera are user manageable, not motion sensors. Friday, February 22, 2019

Problems We can’t turn off motion sensors. All applications have access to motion sensors by default. Permissions allows control of access to data directly sensed by the sensors, but not to information that can be inferred indirectly from the sensors! Smartwatch motions sensors are more difficult to manage than on smartphone. Similar to smartphones motion sensors are nor user manageable. Friday, February 22, 2019

Our Previous Work “(Smart)Watch Your Taps” ISWC’15 Friday, February 22, 2019

New Target: QWERTY Keyboards Friday, February 22, 2019

Linear Accelerometer Readings We Asked Ourselves Is it Possible to Infer What is Being Typed on the Keyboard Based on the Wrist Movements Observable by the Smartwatch Motion Sensors? We observed that there was more motion when ever a key on the left side of the keyboard was pressed. Smartwatch worn on left hand. Q M Linear Accelerometer Readings Friday, February 22, 2019

Dividing the Keyboard Thus, an attacker can infer which side of the keyboard was pressed based on the level of activity during a key press. Friday, February 22, 2019

Assuming watch is worn on left hand Further Observations We can also categorize the direction of movement for the watch wearing hand. Normally we type multiple characters at a time, such as words or sentences. Therefore, analyzing pairs of keystrokes can fetch us more information than treating them individually. So, we categorize key pairs based on geographical directions such as N, S….And O represents the same key presses consecutively. Assuming watch is worn on left hand Friday, February 22, 2019

Forming “Word-Profiles” Word-profile for the word “boards”: bo oa ar rd ds R X R . R X L . L E L . L S L . L W L X represent unknown transitions, where right hand alphabets are involved in the character pair. Mention about this challenge, where we don’t have information from one hand. Friday, February 22, 2019

Learning Phase One of the authors typed the training words. Friday, February 22, 2019

Attack Phase Friday, February 22, 2019

Samsung Gear Live smartwatch Evaluation 25 participants aged between 19-32 years. Matlab and PyBrain Matlab for feature extraction. PyBrain for machine learning. Samsung Gear Live smartwatch Anker A7726121 Bluetooth keyboard Friday, February 22, 2019

Results: Basic Text Recovery Dictionary: Ten sentences in List 6 of Harvard sentences Typed: The same ten sentences above L-R classifier misclassifications: 0% N-E-S-W-O classifier misclassifications: ~5% Word Recovery Error: Out of 48 words of four letters or more, only 3 were not recovered correctly (93.75% success in recovery) Friday, February 22, 2019

Similarity Score Closest Matching Word-Profile Frequency of Use Based on number of matching features Frequency of Use As in Dictionary Pool or English Literature Similarity score is checked against all words in the dictionary. Friday, February 22, 2019

Results: Contextual Dictionary Participants typed a paragraph of 40 words (of length four or more) that appear in a National Public Radio (NPR) news article on Greece debt crisis, and this experiment simulates eavesdropping on a reporter typing the NPR news article. The dictionary is formed with words that appear in six other news articles related to Greece debt crisis, that were published a week before the target article. Friday, February 22, 2019

Results: Contextual Dictionary Contextual Dictionary: Percentage of words recovered per participant, presented in descending order of typing speed of the participants. Friday, February 22, 2019

Results: Typing Behavior and Speed We observed that in many instances participants did not follow our assumed layout. Some of the participants frequently used their left hand to press a key on the right side of the keyboard, and vice versa. We also found that participant who typed slower, were less likely to follow the left and right division of the keyboard. Friday, February 22, 2019

Results: Typing Behavior and Speed Talk about attacker’s speed compared to participants. Contextual Dictionary: Percentage of words recovered per participant, presented in descending order of typing speed of the participants. Friday, February 22, 2019

Results: Large Dictionary 38 English words typed by participants. English dictionary of 60,000 words, sorted by frequency of use in English literature. Problem of colliding word-profiles: Show: LXR . RXR . RXL Sums: LXR . RXR . RXL Explain colliding word-profiles. Friday, February 22, 2019

Results: Large Dictionary This is similar to Google search (or Baidu). If the desired search result is on the first page, it is more useful than having to search among 20 pages. We know no one goes to Google second page! Talk about drawbacks in experimental setup of Marquardt et al. [surface vibration] and Berger et al. [acoustic emanation]. A comparison of accuracy of our attack with Marquardt et al. [surface vibration] and Berger et al. [acoustic emanation]. Note that in spite of not having wrist movement information available from the non-watch-wearing hand, our results are roughly comparable for a large (60,000 words) dictionary. Friday, February 22, 2019

Limitations Ambient Wrist Movement Left and Right Handedness. But… Inferring Non-Dictionary Text Friday, February 22, 2019

Smart Mitigation Access control over seemingly innocuous sensors required. But should not be the old-fashioned way. Must be context-aware in order to automatically manage sensor permissions, without having the user to manually change these settings repetitively. Explain poor usability in ACL. Can we cut-off the motion sensors when the user is typing? Why not? So we used the same motion sensors to develop a typing activity detection framework. Friday, February 22, 2019

Proposed Protection Framework (i) a real-time typing activity detection (rTAD) and (ii) a motion sensor access-controller (MSAC) Energy: Activity measured in terms of cumulative linear accelerometer readings. An unworn watch lying on a table has zero energy, while an athlete's watch has high energy. Typing activity typically results in low but nonzero energy. We apply a low pass filter over the linear accelerometer to eliminate high-frequency noise caused by environmental factors. Turnarounds: Major positive to negative (or vice versa) changes on linear accelerometer readings signify the turnarounds adjoining transitional movements between key presses. Multiple turnarounds in close time proximity can be associated with many activities, such as brushing teeth, eating, playing drums, etc. As a result, we need additional features to distinguish typing from other similar activities. Magnetic Field Change: Wrists are not rotated significantly when a user types on a QWERTY keyboard, while sitting in front of a stationary desk. Rapid change in north, east and nadir vectors implies non-typing activity. Direction of Gravity: Gravity generally remains dominant on z-axis of accelerometer while typing on a horizontally placed keyboard. Any major fluctuations or gravity on x-axis or y-axis implies other activities. Step Count: We assume that the user will be stationary while typing on a computer keyboard. Thus, whenever step count increases, we rule out typing activity. Friday, February 22, 2019

rTAD Parameters Energy: Activity measured in terms of cumulative linear accelerometer readings. An unworn watch lying on a table has zero energy, while an athlete's watch has high energy. Typing activity typically results in low but nonzero energy. We apply a low pass filter over the linear accelerometer to eliminate high-frequency noise caused by environmental factors. Turnarounds: Major positive to negative (or vice versa) changes on linear accelerometer readings signify the turnarounds adjoining transitional movements between key presses. Multiple turnarounds in close time proximity can be associated with many activities, such as brushing teeth, eating, playing drums, etc. As a result, we need additional features to distinguish typing from other similar activities. Magnetic Field Change: Wrists are not rotated significantly when a user types on a QWERTY keyboard, while sitting in front of a stationary desk. Rapid change in north, east and nadir vectors implies non-typing activity. Direction of Gravity: Gravity generally remains dominant on z-axis of accelerometer while typing on a horizontally placed keyboard. Any major fluctuations or gravity on x-axis or y-axis implies other activities. Step Count: We assume that the user will be stationary while typing on a computer keyboard. Thus, whenever step count increases, we rule out typing activity. Friday, February 22, 2019

Motion Sensor Access-Controller (MSAC) Complete Blocking Reduced Sampling Rate Random Out of Order Blocks Explain positives and negatives of each. We were not able to implement it, but should be easy task for OS developers to implement. Friday, February 22, 2019

rTAD Evaluation: High Sensitivity Friday, February 22, 2019

rTAD Evaluation: Low Sensitivity Friday, February 22, 2019

rTAD Evaluation Results Results can be further improved with new features. Friday, February 22, 2019

Conclusion A new keystroke inference attack which utilizes wrist-motion data gathered from a smartwatch as side-channel information. A smart protection framework to detect typing activity and automatically regulate sensor access, aimed to improve privacy without degrading utility of the device. Thank You! http://sprite.cs.wichita.edu/ Friday, February 22, 2019