September Activity Recognition and Biometric Identification Using Cell Phone Accelerometers WISDM Project Department of Computer & Info. Science Fordham University

September We are Interested in WISDM WISDM: WIreless Sensor Data Mining WISDM: WIreless Sensor Data Mining Powerful portable wireless devices are becoming common and are filled with sensors Powerful portable wireless devices are becoming common and are filled with sensors Smart phones: Android phones, iPhone Smart phones: Android phones, iPhone Music players: iPod Touch Music players: iPod Touch Sensors on smart phones include: Sensors on smart phones include: Microphone, camera, light sensor, proximity sensor, temperature sensor, GPS, compass, accelerometer Microphone, camera, light sensor, proximity sensor, temperature sensor, GPS, compass, accelerometer

WISDM Data Mining Problems Completed initial stages of research on 2 tasks: Completed initial stages of research on 2 tasks: Activity Recognition Activity Recognition What is the user doing? What is the user doing? Biometric Identification Biometric Identification Who is the user? Is the user who they claim to be? Who is the user? Is the user who they claim to be? Future tasks Future tasks GPS mining: learn about user routes & interactions GPS mining: learn about user routes & interactions Use cell phones as a sensor network to learn about the environment Use cell phones as a sensor network to learn about the environment September

4 Accelerometer-Based Activity Recognition The Problem: use accelerometer data to determine a user’s activity The Problem: use accelerometer data to determine a user’s activity Activities include: Activities include: Walking and jogging Walking and jogging Sitting and standing Sitting and standing Ascending and descending stairs Ascending and descending stairs More activities to be added in future work More activities to be added in future work

September Applications of Activity Recognition Health Applications Health Applications Generate activity profile to monitor overall type and quantity of activity Generate activity profile to monitor overall type and quantity of activity Parents can use it to monitor their children Parents can use it to monitor their children Can be used to monitor the elderly Can be used to monitor the elderly Make the device context-sensitive Make the device context-sensitive Cell phone sends all calls to voice mail when jogging Cell phone sends all calls to voice mail when jogging Adjust music based on the activity Adjust music based on the activity Broadcast (Facebook) your every activity Broadcast (Facebook) your every activity

Accelerometer-Based Biometric Identification The Problem: Use accelerometer data to identify an individual user The Problem: Use accelerometer data to identify an individual user Identity prediction: map a user to one of a set of predetermined users Identity prediction: map a user to one of a set of predetermined users Authentication: determine whether a user is who they claim to be Authentication: determine whether a user is who they claim to be September

Applications of Biometric Identification Security & theft prevention of mobile devices Security & theft prevention of mobile devices Automatic personalization of mobile devices Automatic personalization of mobile devices For example, send all calls to voic when jogging For example, send all calls to voic when jogging Identify user and load proper settings Identify user and load proper settings General Security Applications General Security Applications Should the user be in this location? Should the user be in this location? Can be used as a second level of security Can be used as a second level of security September

8 Our WISDM Platform Platform based on Android cell phones Platform based on Android cell phones Android is Google’s open source mobile computing OS Android is Google’s open source mobile computing OS Easy to program, free, will have a large market share Easy to program, free, will have a large market share Android phones now outselling iPhones Android phones now outselling iPhones Unlike most other work on activity recognition: Unlike most other work on activity recognition: No specialized equipment No specialized equipment Single device naturally placed on body (in pocket) Single device naturally placed on body (in pocket)

September Our WISDM Platform Current research was conducted off-line Current research was conducted off-line Data was collected and later analyzed off-line Data was collected and later analyzed off-line Now updating our platform to operate in real-time Now updating our platform to operate in real-time In June we released real-time sensor data collection app to Android marketplace In June we released real-time sensor data collection app to Android marketplace Currently collects accelerometer and GPS data and transmits it to our server Currently collects accelerometer and GPS data and transmits it to our server

September Accelerometers Included in most smart phones & other devices Included in most smart phones & other devices All Android phones, iPhones, iPod Touches, etc. All Android phones, iPhones, iPod Touches, etc. Tri-axial accelerometers that measure 3 dimensions Tri-axial accelerometers that measure 3 dimensions Initially included for screen rotation and advanced game play Initially included for screen rotation and advanced game play

September Examples of Raw Data Next few slides show data for one user over a few seconds for various activities Next few slides show data for one user over a few seconds for various activities Cell phone is in user’s pocket Cell phone is in user’s pocket Earth’s gravity is registered as acceleration Earth’s gravity is registered as acceleration Acceleration values relative to axes of the device, not Earth Acceleration values relative to axes of the device, not Earth In theory we can correct this given that we can determine orientation of the device In theory we can correct this given that we can determine orientation of the device

September Standing

September Sitting

September Walking

September Jogging

September Descending Stairs

September Ascending Stairs

September Data Collection Procedure User’s move through a specific course User’s move through a specific course Perform various activities for specific times Perform various activities for specific times Data collected using Android phones Data collected using Android phones Activities labeled using our Android app Activities labeled using our Android app Data collection procedure approved by Fordham Institutional Review Board (IRB) Data collection procedure approved by Fordham Institutional Review Board (IRB) Collected data from ~35 users (will increase) Collected data from ~35 users (will increase)

September Data Preprocessing Convert time series data into examples so we can use standard classifiers (e.g., decision trees) Convert time series data into examples so we can use standard classifiers (e.g., decision trees) Use a 10 second example duration/window Use a 10 second example duration/window 3 acceleration values every 50 ms (600 total values) 3 acceleration values every 50 ms (600 total values) Generate 43 total features Generate 43 total features Ave. acceleration each axis (3) Ave. acceleration each axis (3) Standard deviation each axis (3) Standard deviation each axis (3) Binned/histogram distribution for each axis (30) Binned/histogram distribution for each axis (30) Time between peaks (3), Ave resultant acceleration (1) Time between peaks (3), Ave resultant acceleration (1)

ACTIVITY RECOGNITION RESULTS September

September Activity Recognition Final Data Set

September Data Mining Step Utilized three WEKA learning methods Utilized three WEKA learning methods Decision Tree (J48) Decision Tree (J48) Logistic Regression Logistic Regression Neural Network Neural Network Results reported using 10-fold cross validation Results reported using 10-fold cross validation

September Summary Results

September J48 Confusion Matrix Predicted Class WalkJogUpDownSitStand ActualClassActualClass Walk Jog Up Down Sit Stand

September Activity Recognition Conclusions Able to identify activities with good accuracy Able to identify activities with good accuracy Hard to differentiate between ascending and descending stairs. To limited degree also looks like walking. Hard to differentiate between ascending and descending stairs. To limited degree also looks like walking. Can accomplish this with a cell phone placed naturally in pocket Can accomplish this with a cell phone placed naturally in pocket Accomplished with simple features and standard data mining methods Accomplished with simple features and standard data mining methods

BIOMETRIC IDENTIFICATION RESULTS September

Biometric Identification Data Sets We evaluated 6 data sets (4 activities) We evaluated 6 data sets (4 activities) Aggregate (all 6 activities without class labels) Aggregate (all 6 activities without class labels) Walking Walking Jogging Jogging Ascending Stairs Ascending Stairs Descending Stairs Descending Stairs Aggregate-Oracle (all 6 activities with class labels) Aggregate-Oracle (all 6 activities with class labels) The unlabeled aggregate data set is most realistic The unlabeled aggregate data set is most realistic September

# of Examples per User/Activity September

Accuracy for Person Identification (based on 10-Second Examples) September J48: Decision Tree Learner Straw Man: Strategy of always predicting the most common user

Aggregate Data Set Confusion Matrix September (Results for first 14 users only)

We Can Do Better: Majority Scheme We know which records come from the same cell phone user We know which records come from the same cell phone user So predict the users identity based on the identity predicted most often So predict the users identity based on the identity predicted most often September

Results Using Majority Voting September

Ratio of Records Correctly Classified to Most Successful Imposter September

Authentication Results Positive authentication rate: % of test examples coming from a user that are correctly classified as belonging to that user Negative authentication rate: % of test examples from an imposter that are correctly identified as not belonging to the user September

Biometric Identification Conclusions Very accurate models for person identification using data mining of accelerometer data Very accurate models for person identification using data mining of accelerometer data Generally perfect performance for identification when using majority scheme Generally perfect performance for identification when using majority scheme Can get good biometric results without knowing the specific activity the user is performing Can get good biometric results without knowing the specific activity the user is performing September

September Related Work At least a dozen papers on activity recognition using multiple sensors, mainly accelerometers At least a dozen papers on activity recognition using multiple sensors, mainly accelerometers Typically studies only users Typically studies only users Activity recognition also done via computer vision Activity recognition also done via computer vision Actigraphy uses devices to study movement Actigraphy uses devices to study movement Used by psychologists to study sleep disorders, ADD Used by psychologists to study sleep disorders, ADD A few recent efforts use cell phones A few recent efforts use cell phones Yang (2009) used Nokia N95 and 4 users Yang (2009) used Nokia N95 and 4 users Brezmes (2009) used Nokia N95 with real-time recognition Brezmes (2009) used Nokia N95 with real-time recognition One model per user (requires labeled data from each user) One model per user (requires labeled data from each user)

September Future Work Add more activities and users Add more activities and users Add more sophisticated features Add more sophisticated features Try time-series based learning methods Try time-series based learning methods Deploy higher level applications: activity profiler Deploy higher level applications: activity profiler Can be used to encourage healthy behaviors Can be used to encourage healthy behaviors Can benefit the young and the elderly Can benefit the young and the elderly

Future Work (cont.) Generate results in real time Generate results in real time Activity recognition Activity recognition Uses a universal model so no need to train per user Uses a universal model so no need to train per user Send results to server and get response back and on Web Send results to server and get response back and on Web Alternatively do everything on the phone Alternatively do everything on the phone Biometric Identification Biometric Identification Need a model per user so would need to train model Need a model per user so would need to train model Not too hard, just collect unlabelled activity data Not too hard, just collect unlabelled activity data September

Future Work (cont.) GPS Data Mining GPS Data Mining Find spatio-temporal patterns from GPS data Find spatio-temporal patterns from GPS data Location based on time of day, day of week Location based on time of day, day of week Identify friends and people you spend time with Identify friends and people you spend time with Identify things about the environment Identify things about the environment Where are the pedestrian paths on campus? Where are the pedestrian paths on campus? How busy is the cafeteria and when? How busy is the cafeteria and when? Where do people congregate Where do people congregate … September

For More Information See the WISDM web site: See the WISDM web site: See the two published papers on activity recognition and biometric identification See the two published papers on activity recognition and biometric identification Meetings usually Thursday at 6:30 pm Meetings usually Thursday at 6:30 pm Talk to me Talk to me There are quite a few benefits to undergraduate research! There are quite a few benefits to undergraduate research! September

September Thank You Questions?