1. TOUCHSIGNATURES Maryam Mehrnezhad, Ehsan Toreini, Siamak F. Shahandashti, Feng Hao Newcastle University CryptoForma meeting, Belfast 4 May 2015

2. The Attack Identification of User Touch Actions Based on Mobile Sensor Data via JavaScript Accepted in ASIACCS’15 Touch ActionDescription ClickTouching an item momentarily with one finger Scroll (Up, Down, Right, Left)Touching continuously and simultaneously sliding in the corresponding direction Zoom (In, Out)Placing 2 fingers on the screen and sliding them apart or toward each other, respectively HoldTouching continuously for a while with one finger

3. HTML 5 HTML5 is moving toward handling system functionalities: Ideas such as B2G(boot2gecko) by Mozilla Having that in mind, it is not surprising that HTML5 can get mobile sensor related data

4. HTML 5 Currently, mobile web applications have access to the following sensors: Geolocation Multimedia (video, camera, microphone, webcams) Ambient light motion and orientation

5. HTML 5 According to W3C specifications, modern web browsers allow JavaScript code to access motion and orientation sensor data.

6. Core IDEA This project targets this question: What are the possible privacy leakages? Is it possible to recognize user actions using the sensor data acquired by JavaScript? Neither iOS nor Android ask permission to access these sensors via browsers Accessing sensor data within mobile apps has already been studied Different security or privacy attacks in the literature

7. Some Challenges The mobile in-browser sensor data access is only restricted to two streams: Orientation: supplies the physical orientation of the device Device motion: acceleration of the device In-browser access is limited in contrast to raw sensor data access in normal applications: processed data Low rate streams with frequencies around 5 to 10 times slower than in-app data

8. Privacy Breaches Unlike other sensor accesses, no authorization from the user to access orientation and acceleration data. This could possibly leak information such as: User Physical Movements (walking, running, sitting) Some User Interactions with the device that has specific patterns (such as answering calls, Taking Photos) User Touch Actions

9. Touch Actions Identification of touch actions may reveal a range of activities about the user’s interaction with other webpages. E.g. users tend to mostly scroll on a news website while trying mostly to type when using an email client.

10. TouchSignature Our system is able to distinguish user touch actions given access to the device motion and orientation sensor data Attack Model: a malicious web content spying on a user via JavaScript. The content is loaded via an iframe embedded in the webpage. Browser is actively, or passively in the background User has access to the Internet

11. Some Technical Details According to W3C, HTML5 and JavaScript provide access to the following sensor data: Device Orientation: Three rotations, alpha, beta and gamma Device Acceleration: Cartesian coordinates: x,y and z Device Acceleration including Gravity Device Rotation Rate: three rotations alpha, beta and gamma Interval: rate of sensor reading in milliseconds We have developed Touchsignatures: server side as Node.js and mongodB. Client Side JavaScript library called socket.io to send live sensor data streams Use of supervised Machine Learning techniques to analyse data

12. Experiments

13. Touch ActionDescription ClickTouching an item momentarily with one finger Scroll (Up, Down, Right, Left)Touching continuously and simultaneously sliding in the corresponding direction Zoom (In, Out)Placing 2 fingers on the screen and sliding them apart or toward each other, respectively HoldTouching continuously for a while with one finger

15. Experiments We collected data from 11 volunteers We presented each user a brief introduction and instructions to perform 8 touch actions Experiments were performed on google Chrome in iPhone 5. We asked each user to perform each action 5 times Two types of mobile holding were measured: two handed and one handed At the end, we had 10 samples of each touch action for 11 people.

16. Feature Extraction Time Domain Features: Raw Captured Sequence First order derivative of each sequence maximum, minimum and mean of each sequence and its derivative. Total Energy or each sequence and its derivative And some more features, totally 116 features for each touch Frequency Domain: FFT of the sequences Maximum, Minimum, mean and energy of each sequence. Totally 48 features In General, 164 features for each sequence.

17. Classification Process We implemented different classification algorithms: Artificial Neural Networks (ANN) K-Nearest neighbour (k- NN) Decision Tree We used 10 fold cross validation approach 1-NN showed the best performance

18. Phase 1 classification Touch ActionIdentification Rate Click78.18% Hold88.18% Scroll95.91% Zoom In71.82% Zoom Out76.36% Overall87.39%

19. Phase 2 Classification Touch ActionIdentification Rate Scroll Down57.27% Scroll Up69.09% Scroll Right48.18% Scroll Left71.82% Overall61.59%

20. Contribution of Different Sensor Data Streams Orientation has got the best impact in the final results. The rest of the sensor data combined only effects 3.64%

21. Browser Support Device/mOS/Browser ActiveBackgroundLocked Determines the Device/Mobile OS/ Browser Name Status: When the browser is running actively and interacting with the user Status: When the browser is not active, but running in background Status: When browser is not active and the device screen is locked.

22. Browser Support Device/mOS/Browser ActiveBackgroundLocked SameIntraOther SameOtherSameOther When the webpage visited is manipulated

23. Comparisons of the Popular Browser Sensor Accessibility in Android/iOS

24. Possible Solutions Notify users within browserOperating System Settings

25. Future? Is it possible to recognize the keys has been pressed by using this low rate data? Other privacy breaches? Open to any other suggestions…

26. Conclusion First to perform a practical privacy attack by Sensor data using JavaScript User Actions could be recognized by using this sensor data, even if it is processed and provided in low rates. This shows a major shortcoming in mobile operating systems and browser access control policies with respect to user privacy. We suggest to apply the same approach as GPS access by providing effective user notification and control mechanism

27. THANKS!