1. TapPrints: Your Finger Taps Have Fingerprints Emiliano Miluzzo*, Alex Varshavsky*, Suhrid Balakrishnan*, Romit R. Choudhury + * at&t Labs – Research, USA + Duke University, USA © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.

2. Vs

3. sensing with smartphones and tablets © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. compass accelerometer gyroscope compass barometer compass location

4. sensing with smartphones and tablets © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.

5. sensing with smartphones and tablets © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.

6. what is sensing used for? © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.

7. what is sensing used for? © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.

8. what is sensing used for? © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.

9. what is sensing used for? © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.

10. however… © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.

14. x 1, y 1

15. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. x 2, y 2

16. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. x 3, y 3

17. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.

20. facts motion sensors do not require user permission motion sensors can run in the background W3C proposal for motion sensor data access with Javascript © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.

21. threat model legitimate apps exploiting motion sensor data for malicious purposes malicious apps secretly installed web app access to motion sensors © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.

22. practical scenario © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.

23. practical scenario © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. accel and gyro data from N people

24. practical scenario © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. accel and gyro data from N people build a model for the keys

25. practical scenario © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. accel and gyro data from N people build a model for the keys deploy app + model

26. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. z

27. p

28. design © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.

30. how do we model the taps so that we can recognize them?

31. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. ML approach accel and gyro data from N people build a model for the keys deploy app + model

32. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. ML approach accel and gyro data from N people build a model for the keys deploy app + model

33. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. ML approach accel and gyro data from N people build a model for the keys deploy app + model

34. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. ML approach accel and gyro data from N people build a model for the keys deploy app + model

35. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. ML approach

36. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. ML approach

37. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. ML approach

38. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. ML approach M-class classifier (M=26 for letters)

39. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. ML approach

40. features © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.

41. features time domain moments, variance, min/max, skewness (for asymmetry of vector components), kurtosis (to measure the ‘peaked-ness’ of vector components) correlation between the 3-axis of both the accelerometer and gyro using different norms first-order numerical derivatives and energy angle between the accelerometer and gyro vector and the rate of change frequency domain fast fourier transform, power, etc. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.

42. learning classifiers kNN, SVM, multinomial logistic regression, random forest, bagged decision tree ensemble method 21 classifiers and the final result is given from the most predicted vote © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.

43. experimental results © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.

45. 10 volunteers ~40.000 labeled taps 4 weeks portrait thumb, portrait index finger, landscape both thumbs They are asked to press icons letters sequentially pangrams © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. methodology

46. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. icon tap experiment

47. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. icon tap experiment random guess inference accuracy

48. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. icon tap experiment

49. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. icon tap experiment

50. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. icon tap experiment

51. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. icon tap experiment

52. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. icon tap experiment

53. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. borrowing strength is a pre-trained model enough to infer taps in the wild?

54. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. borrowing strength given N users, train on N-1 users and test on the left-out user

55. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. borrowing strength

56. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. borrowing strength

57. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. borrowing strength

58. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. borrowing strength

59. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. borrowing strength

60. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. borrowing strength

61. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. borrowing strength

62. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. letter press - tablet

63. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. letter press - tablet random guess inference accuracy

64. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. letter press - tablet

65. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. letter press - tablet

66. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. letter press - tablet

67. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. pangram - tablet

68. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. pangram - tablet the quick brown fox jumps over the lazy dog

69. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. pangram - tablet

70. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. pangram - tablet

71. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. pangram - tablet

72. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. pangram - tablet

73. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. pangram - tablet r, w, s, a

74. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. pangram - tablet

75. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. letter press - phone avg accuracy between 37% and 66%

76. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. pangram - phone

77. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. pangram - phone

78. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. pangram - phone

79. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. pangram - phone dictionary attacks combined with confusion matrix prior could boost the accuracy

80. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. most effective sensor?

81. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. most effective sensor?

82. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. most effective sensor?

83. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. most effective sensor?

84. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. sensor sampling rate plays a role

85. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. possible solutions? block motion sensor data when the keyboard is up reduce motion sensors’ sampling rate for an app running in the background review the W3C draft for motion sensor access through Javascript?

86. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. possible solutions?

87. © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. future work exploiting tap inference to profile a user’s behavior (to identify the device legitimate user for example)?

88. to tap or not to tap, that is the question… miluzzo@research.att.com © 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.