1. September 2010 1 Activity Recognition and Biometric Identification Using Cell Phone Accelerometers WISDM Project Department of Computer & Info. Science Fordham University

2. September 2010 2 We are Interested in WISDM WISDM: WIreless Sensor Data Mining WISDM: WIreless Sensor Data Mining Powerful portable wireless devices are becoming common and are filled with sensors Powerful portable wireless devices are becoming common and are filled with sensors Smart phones: Android phones, iPhone Smart phones: Android phones, iPhone Music players: iPod Touch Music players: iPod Touch Sensors on smart phones include: Sensors on smart phones include: Microphone, camera, light sensor, proximity sensor, temperature sensor, GPS, compass, accelerometer Microphone, camera, light sensor, proximity sensor, temperature sensor, GPS, compass, accelerometer

3. WISDM Data Mining Problems Completed initial stages of research on 2 tasks: Completed initial stages of research on 2 tasks: Activity Recognition Activity Recognition What is the user doing? What is the user doing? Biometric Identification Biometric Identification Who is the user? Is the user who they claim to be? Who is the user? Is the user who they claim to be? Future tasks Future tasks GPS mining: learn about user routes & interactions GPS mining: learn about user routes & interactions Use cell phones as a sensor network to learn about the environment Use cell phones as a sensor network to learn about the environment September 2010 3

4. 4 Accelerometer-Based Activity Recognition The Problem: use accelerometer data to determine a user’s activity The Problem: use accelerometer data to determine a user’s activity Activities include: Activities include: Walking and jogging Walking and jogging Sitting and standing Sitting and standing Ascending and descending stairs Ascending and descending stairs More activities to be added in future work More activities to be added in future work

5. September 2010 5 Applications of Activity Recognition Health Applications Health Applications Generate activity profile to monitor overall type and quantity of activity Generate activity profile to monitor overall type and quantity of activity Parents can use it to monitor their children Parents can use it to monitor their children Can be used to monitor the elderly Can be used to monitor the elderly Make the device context-sensitive Make the device context-sensitive Cell phone sends all calls to voice mail when jogging Cell phone sends all calls to voice mail when jogging Adjust music based on the activity Adjust music based on the activity Broadcast (Facebook) your every activity Broadcast (Facebook) your every activity

6. Accelerometer-Based Biometric Identification The Problem: Use accelerometer data to identify an individual user The Problem: Use accelerometer data to identify an individual user Identity prediction: map a user to one of a set of predetermined users Identity prediction: map a user to one of a set of predetermined users Authentication: determine whether a user is who they claim to be Authentication: determine whether a user is who they claim to be September 2010 6

7. Applications of Biometric Identification Security & theft prevention of mobile devices Security & theft prevention of mobile devices Automatic personalization of mobile devices Automatic personalization of mobile devices For example, send all calls to voicemail when jogging For example, send all calls to voicemail when jogging Identify user and load proper settings Identify user and load proper settings General Security Applications General Security Applications Should the user be in this location? Should the user be in this location? Can be used as a second level of security Can be used as a second level of security September 2010 7

8. 8 Our WISDM Platform Platform based on Android cell phones Platform based on Android cell phones Android is Google’s open source mobile computing OS Android is Google’s open source mobile computing OS Easy to program, free, will have a large market share Easy to program, free, will have a large market share Android phones now outselling iPhones Android phones now outselling iPhones Unlike most other work on activity recognition: Unlike most other work on activity recognition: No specialized equipment No specialized equipment Single device naturally placed on body (in pocket) Single device naturally placed on body (in pocket)

9. September 2010 9 Our WISDM Platform Current research was conducted off-line Current research was conducted off-line Data was collected and later analyzed off-line Data was collected and later analyzed off-line Now updating our platform to operate in real-time Now updating our platform to operate in real-time In June we released real-time sensor data collection app to Android marketplace In June we released real-time sensor data collection app to Android marketplace Currently collects accelerometer and GPS data and transmits it to our server Currently collects accelerometer and GPS data and transmits it to our server

10. September 2010 10 Accelerometers Included in most smart phones & other devices Included in most smart phones & other devices All Android phones, iPhones, iPod Touches, etc. All Android phones, iPhones, iPod Touches, etc. Tri-axial accelerometers that measure 3 dimensions Tri-axial accelerometers that measure 3 dimensions Initially included for screen rotation and advanced game play Initially included for screen rotation and advanced game play

11. September 2010 11 Examples of Raw Data Next few slides show data for one user over a few seconds for various activities Next few slides show data for one user over a few seconds for various activities Cell phone is in user’s pocket Cell phone is in user’s pocket Earth’s gravity is registered as acceleration Earth’s gravity is registered as acceleration Acceleration values relative to axes of the device, not Earth Acceleration values relative to axes of the device, not Earth In theory we can correct this given that we can determine orientation of the device In theory we can correct this given that we can determine orientation of the device

12. September 2010 12 Standing

13. September 2010 13 Sitting

14. September 2010 14 Walking

15. September 2010 15 Jogging

16. September 2010 16 Descending Stairs

17. September 2010 17 Ascending Stairs

18. September 2010 18 Data Collection Procedure User’s move through a specific course User’s move through a specific course Perform various activities for specific times Perform various activities for specific times Data collected using Android phones Data collected using Android phones Activities labeled using our Android app Activities labeled using our Android app Data collection procedure approved by Fordham Institutional Review Board (IRB) Data collection procedure approved by Fordham Institutional Review Board (IRB) Collected data from ~35 users (will increase) Collected data from ~35 users (will increase)

19. September 2010 19 Data Preprocessing Convert time series data into examples so we can use standard classifiers (e.g., decision trees) Convert time series data into examples so we can use standard classifiers (e.g., decision trees) Use a 10 second example duration/window Use a 10 second example duration/window 3 acceleration values every 50 ms (600 total values) 3 acceleration values every 50 ms (600 total values) Generate 43 total features Generate 43 total features Ave. acceleration each axis (3) Ave. acceleration each axis (3) Standard deviation each axis (3) Standard deviation each axis (3) Binned/histogram distribution for each axis (30) Binned/histogram distribution for each axis (30) Time between peaks (3), Ave resultant acceleration (1) Time between peaks (3), Ave resultant acceleration (1)

20. ACTIVITY RECOGNITION RESULTS September 2010 20

21. September 2010 21 Activity Recognition Final Data Set

22. September 2010 22 Data Mining Step Utilized three WEKA learning methods Utilized three WEKA learning methods Decision Tree (J48) Decision Tree (J48) Logistic Regression Logistic Regression Neural Network Neural Network Results reported using 10-fold cross validation Results reported using 10-fold cross validation

23. September 2010 23 Summary Results

24. September 2010 24 J48 Confusion Matrix Predicted Class WalkJogUpDownSitStand ActualClassActualClass Walk151314728220 Jog161275161211 Up882332310722 Down99139225812 Sit40232703 Stand41271208

25. September 2010 25 Activity Recognition Conclusions Able to identify activities with good accuracy Able to identify activities with good accuracy Hard to differentiate between ascending and descending stairs. To limited degree also looks like walking. Hard to differentiate between ascending and descending stairs. To limited degree also looks like walking. Can accomplish this with a cell phone placed naturally in pocket Can accomplish this with a cell phone placed naturally in pocket Accomplished with simple features and standard data mining methods Accomplished with simple features and standard data mining methods

26. BIOMETRIC IDENTIFICATION RESULTS September 2010 26

27. Biometric Identification Data Sets We evaluated 6 data sets (4 activities) We evaluated 6 data sets (4 activities) Aggregate (all 6 activities without class labels) Aggregate (all 6 activities without class labels) Walking Walking Jogging Jogging Ascending Stairs Ascending Stairs Descending Stairs Descending Stairs Aggregate-Oracle (all 6 activities with class labels) Aggregate-Oracle (all 6 activities with class labels) The unlabeled aggregate data set is most realistic The unlabeled aggregate data set is most realistic September 2010 27

28. # of Examples per User/Activity September 2010 28

29. Accuracy for Person Identification (based on 10-Second Examples) September 2010 29 J48: Decision Tree Learner Straw Man: Strategy of always predicting the most common user

30. Aggregate Data Set Confusion Matrix September 2010 30 (Results for first 14 users only)

31. We Can Do Better: Majority Scheme We know which records come from the same cell phone user We know which records come from the same cell phone user So predict the users identity based on the identity predicted most often So predict the users identity based on the identity predicted most often September 2010 31

32. Results Using Majority Voting September 2010 32

33. Ratio of Records Correctly Classified to Most Successful Imposter September 2010 33

34. Authentication Results Positive authentication rate: % of test examples coming from a user that are correctly classified as belonging to that user Negative authentication rate: % of test examples from an imposter that are correctly identified as not belonging to the user September 2010 34

35. Biometric Identification Conclusions Very accurate models for person identification using data mining of accelerometer data Very accurate models for person identification using data mining of accelerometer data Generally perfect performance for identification when using majority scheme Generally perfect performance for identification when using majority scheme Can get good biometric results without knowing the specific activity the user is performing Can get good biometric results without knowing the specific activity the user is performing September 2010 35

36. September 2010 36 Related Work At least a dozen papers on activity recognition using multiple sensors, mainly accelerometers At least a dozen papers on activity recognition using multiple sensors, mainly accelerometers Typically studies only 10-20 users Typically studies only 10-20 users Activity recognition also done via computer vision Activity recognition also done via computer vision Actigraphy uses devices to study movement Actigraphy uses devices to study movement Used by psychologists to study sleep disorders, ADD Used by psychologists to study sleep disorders, ADD A few recent efforts use cell phones A few recent efforts use cell phones Yang (2009) used Nokia N95 and 4 users Yang (2009) used Nokia N95 and 4 users Brezmes (2009) used Nokia N95 with real-time recognition Brezmes (2009) used Nokia N95 with real-time recognition One model per user (requires labeled data from each user) One model per user (requires labeled data from each user)

37. September 2010 37 Future Work Add more activities and users Add more activities and users Add more sophisticated features Add more sophisticated features Try time-series based learning methods Try time-series based learning methods Deploy higher level applications: activity profiler Deploy higher level applications: activity profiler Can be used to encourage healthy behaviors Can be used to encourage healthy behaviors Can benefit the young and the elderly Can benefit the young and the elderly

38. Future Work (cont.) Generate results in real time Generate results in real time Activity recognition Activity recognition Uses a universal model so no need to train per user Uses a universal model so no need to train per user Send results to server and get response back and on Web Send results to server and get response back and on Web Alternatively do everything on the phone Alternatively do everything on the phone Biometric Identification Biometric Identification Need a model per user so would need to train model Need a model per user so would need to train model Not too hard, just collect unlabelled activity data Not too hard, just collect unlabelled activity data September 2010 38

39. Future Work (cont.) GPS Data Mining GPS Data Mining Find spatio-temporal patterns from GPS data Find spatio-temporal patterns from GPS data Location based on time of day, day of week Location based on time of day, day of week Identify friends and people you spend time with Identify friends and people you spend time with Identify things about the environment Identify things about the environment Where are the pedestrian paths on campus? Where are the pedestrian paths on campus? How busy is the cafeteria and when? How busy is the cafeteria and when? Where do people congregate Where do people congregate … September 2010 39

40. For More Information See the WISDM web site: See the WISDM web site: http://storm.cis.fordham.edu/~gweiss/wisdm/ http://storm.cis.fordham.edu/~gweiss/wisdm/ See the two published papers on activity recognition and biometric identification See the two published papers on activity recognition and biometric identification Meetings usually Thursday at 6:30 pm Meetings usually Thursday at 6:30 pm Talk to me Talk to me There are quite a few benefits to undergraduate research! There are quite a few benefits to undergraduate research! September 2010 40

41. September 2010 41 Thank You Questions?