Recognizing Daily Routines Through Activity Spotting Ulf Blanke and Bernt Schiele Computer Science Department, TU Darmstadt

Activity Recognition  Human Activity Recognition  support awareness of applications (human to application)  support analysis of human activity (human to human)  Application Areas  Healthcare: long term monitoring of patients (months!)  Elderly care: personal diary (weeks to months)  Industrial: workshop activities (minutes to a hours) Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 2

Towards High Level Activities Low Level Activities  E.g., walking, standing, biking…  Lasting from seconds to minutes  Detected by pose or characteristic motion. High Level Activities  E.g., daily routines: morning routine, dinner, working…  More important for many domains (e.g., healthcare)  Lasting from minutes to hours  Consist of multiple low level activities: approaching car (walking) driving leaving car (walking) commuting preparing food eating doing dishes dinner …… … … Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 3

Related Work Multilayer Approaches commuting, working, lunch, dinner… High-level activities Low level activities Sensor data Zhang, D., Gatica-Perez, D., Bengio, S., McCowan, I., Lathoud, G. (CVPR 2004) Clarkson, B., Pentland, A. (ICASSP 1999) Oliver, N., Horvitz, E., Garg, A.: (Multimodal Interfaces 2002) Mid-level activities Morning Routine wash undressing drying Cleaning teeth scrub e.g. walking, running, standing, sitting shower Putting toothpaste drying... - Many parameters - Computationally intensive - Many parameters - Computationally intensive Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 4

Related Work Direct Approach commuting, working, lunch, dinner… High-level activities Low level activities Sensor data Mid-level activities e.g. walking, running, standing, sitting Huynh, T., Blanke, U., Schiele, B. (LoCA 2007) Morning Routine - High level activities exhibit high inner-class variability - All Data has to be considered - High level activities exhibit high inner-class variability - All Data has to be considered Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 5

 Can we learn distinctive parts of high level activities?  Can we reduce the amount of data important for recognition? Research Questions Activity Spotting Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 6

Research Questions Activity Spotting Lunch High-level activities Low level activities Sensor data Which low level parts are important for high level activities? Automatic selection Dinner walking picking up food eating Prep food eating Doing dishes Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 7

Research Questions Activity Spotting Lunch High-level activities Low level activities Sensor data Recognizing high level activities by activity spotting feasible? Dinner walking picking up food eating Prep food eating Doing dishes Activity Spotting Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 8

Method High-level activities Low level activities Sensor data Doing dishes K-means clusters Joint boosting Feature-Calculation Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 9

Low Level Activity Selection (Joint)Boosting (1) Combination of low level activities to infer high-level activities (2) Automatic Selection of most discriminative low level activities (3) Sharing features (i.e. low level activities) across high level activities others Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 10

Experiment Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 11

Experimental Setup Evaluation Metrics  Quantitative Analysis  Tradeoff between precision & recall and  number of low-level selected activities?  how much data is needed (occurrence of activities used)?  Qualitative Analysis  Which activities are used – do they make sense? Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 12

 7 days of a life from a single person [Huynh, T. - Ubicomp ‘08]  Two layers of annotation  4 high level routines, more than 20 low level routines Experimental Setup Dataset Pocket Wrist Commuting Working Dinner Lunch 2 acceleration sensors walking standing in line having a coffee Lunch walking eating Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 13

High-level activities Low level activities Sensor data Doing dishes Feature-Calculation Experimental Setup Fixed Parameters K-means clusters Joint boosting Mean and Variance - over 0.4s window - on (x,y,z)-acceleration - of pocket and wrist Histograms -over 30min window Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 14

High-level activities Low level activities Sensor data Doing dishes Feature-Calculation Experimental Setup Varied Parameters K-means clusters Joint boosting -rounds -Routines’ annotation Kmean centers -soft and hard - K = 60 Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 15

Quantitative Results Soft Assignments Rounds41080Huynh 08 Precision in % Recall in % lowlevel activities in% how much data? In % Hard Assignments Number of lowlevel activities (clusters) How much data Rounds Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 16 Tradeoff

Results Classification scores for one day Reducing number Low level activities Precision loss at borders Scores 80 rounds 10 rounds 4 rounds Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 17

Dinner CommuteLunch sitting/desk activities (47.24%) driving car (32.90%), driving car (21.71%) Time walking (99.23%) sitting / desk activities (97.86%) walking (96.09%) driving bike (47.86%) walking (22.51%) picking up food (16.81%) queuing in line (43.86%) picking up food (14.59%) driving bike (16.76%) sitting/desk activities (31.20%) Lunch Work Commute Dinner Time Distribution of low level label for each selected low level cluster Qualitative Analysis Which activities are used? Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 18

Dinner CommuteLunch sitting/desk activities driving car Time walking sitting / desk activities walking driving bike walking picking up food queuing in line picking up food driving bike sitting/desk activities Lunch Work Commute Dinner Time Distribution of low level label for each selected low level cluster Qualitative Analysis Which activities are used? Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 19

Summary & Conclusion  Can we learn distinctive parts of high level activities? Yes.  Automatic Selection of important data  Top down perspective  Find discriminative parts of a high level routines  Can we reduce the amount of data used for recognition? Yes.  Fraction (~5-8%) of data sufficient to recognize daily routines (~80%)  Filter insignificant data  reduce memory usage and computational costs  suited for embedded long term activity recognition Activity Spotting feasible for routine recognition.  Outperforms previous generative approaches on this dataset [Huynh - Ubicomp 08] Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 20

Thank you for your attention. Questions? ご清聴、ありがとうございました 。 ご質問はありますか。 Dataset available at Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 21

Date | Department of Computer Science | Ulf Blanke | 22 End of Presentation

(Joint)Boosting Weak Classifier b: confidence, that sample is not part of class a: confidence, that sample is part of class weak classifier a b Total confidence of separation Strong classifier Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 23