Sensor Analysis – Part II A literature based exploration Thomas Plötz [material taken from the original papers and John Krumm “Ubiquitous Computing Fundamentals”

ApplicationModalityAnalysis ApproachEvaluation

ApplicationModalityAnalysis ApproachEvaluation

Application Activities of Daily Living kitchen General household sports Health Fall detection Rehab diagnosis Professional / Training manufacturing Quality control HCI User interfaces Accessibility (ASL) Device handling sports: training / reviewing movements after the “fact” daily routine: efficiency (where do you waste time?) – quantified self movement

Activities of Daily Living vanKasteren2008-AARvanKasteren2010-TKOPatterson2005-FGA Perkowitz2004-MMO

ADL Logan2007-ALT

Application Ward2006-ARO

Application ADL well studied and recognized segmentation + classification variety of application domains (health, smart homes, …) - no fine-grained analysis (read: trivial? Beyond proof-of-concept?) - no quality assessment (How good? vs. What?) … your conclusions here!

ApplicationModalityAnalysis ApproachEvaluation

(Sensing) Modality vanKasteren Binary wall-mounted sensors Reed-switches, motion- detectors etc. Logan2007-ALT (placelab) Reed switches Electric current flow sensors Temperature sensors Humidity sensors Light sensors Barometric pressure sens. Gas sensor Water flow sensors Motion detectors (for object use detection) Accelerometers (body- worn) Patterson2005-FGA & Perkowitz2004-MMO RFID (glove-mounted reader, tags on objects) Ward2006-ARO Body-worn microphones Body-worn accelerometers

(Sensing) Modality inexpensive non-obtrusive, privacy respecting (typically …) even wearable - overkill? … your conclusions here!

ApplicationModalityAnalysis ApproachEvaluation

Analysis Approach: Ward2006-ARO Activity Recognition of Assembly Tasks Using Body-Worn Microphones and Accelerometers classical Activity Recognition – ISWC-style

Analysis Approach: Ward2006-ARO

Sound Intensity Analysis for Segmentation of Activities – analysis of two microphone signals (I 1 – wrist ; I 2 – upper arm) Sound Classification (which tool used?) based on spectral analysis and templates (after smoothing)

Analysis Approach: Ward2006-ARO Spectral representation of acceleration data, calculated using sliding window technique HMM-based classification of acceleration data different fusion techniques for integrating hypotheses from acoustic acoustic recognition and accelerator-based processing

Analysis Approach: Ward2006-ARO pioneering work combination of modalities explicit segmentation step focus on classification techniques and fusion (NOT on feature representation)

Analysis Approach: Patterson2005-FGA Fine-Grained Activity Recognition by Aggregating Abstract Object Usage object-centered sensing and recognition (in contrast to ISWC-style)

Analysis Approach: Patterson2005-FGA Independent HMMsconnected HMMsObject-centered HMMs [DBN with aggregates]

Analysis Approach: Patterson2005-FGA

Analysis Approach: vanKasteren2010-TKO Transferring Knowledge of Activity Recognition across Sensor Networks What if we change the environment?

Analysis Approach: vanKasteren2010-TKO list of ADLs is the same for each house, but sensor networks differ 1.How to deal with differences in sensor networks (due to different layouts of houses)? 2.How to learn model parameters that respect differences in behavior of the inhabitants?

Analysis Approach: vanKasteren2010-TKO

Modeling approach: HMM “A prior distribution is learned from the source houses and used to provide a sensible initial value for the model parameters of the target house.” conjugate prior: posterior is of same functional form as prior procedure: 1.learn model parameters for source house (using ML) 2.Learn hyperparameters for initial state, and transition is straightforward using numerical estimation 3.Learn hyperparameters for observations requires feature space mapping and numerical estimation

Analysis Approach: vanKasteren2010-TKO Finally: EM-based estimation of target model parameters exploiting estimated priors

Analysis Approach: Perkowitz2004-MMO Mining Models of Human Activities from the Web What if we don’t have enough (or none) sample data? “ […] requires models of the activities of interest, but model construction does not scale well: humans must specify low- level details, such as segmentation and feature selection of sensor data, and high-level structure, such as spatio- temporal relations between states of the model, for each and every activity.”

Analysis Approach: Perkowitz2004-MMO “[…] we show how to mine very large libraries of human activities from the web, instead of analyzing sensor data.” “PROACT assumes that “interesting” objects in the environment contain RFID tags.” “Users employ RFID tag readers to track tag objects they interact with.” “As users go about their daily activities, the readers detect tags that (a) users touch, (b) are close to them, or (c) are moved by them, and thereby deduce which objects are currently involved in an activity. PROACT uses the sequence and timing of object involvement to deduce what activity is happening.”

Analysis Approach: Perkowitz2004-MMO Modeling based on Hidden (semi) Markov Models general model description via web-mining extract initial parameters (probabilities) using Google conditional probabilities (GCP) P(o, i, l) = GoogleCount(“l”+o) / GoogleCount(l) P(o, i, l): prob. that object o is involved in step i of the activity labeled l.

ApplicationModalityAnalysis ApproachEvaluation

Problem: Skewed distributions (one very dominant class)

Evaluation Problem: Fragmentation; or: Does it really make sense to evaluate sample wise? Idea: Event-based evaluation

ApplicationModalityAnalysis ApproachEvaluation

Bibliography van Kasteren et al. Accurate activity recognition in a home setting. Proc. Int. Conf. on Ubiquitous Computing (2008) Ward et al. Activity recognition of assembly tasks using body-worn microphones and accelerometers. IEEE Trans. Pattern Analysis and Machine Intelligence (2006) vol. 28 (10) pp van Kasteren et al. Transferring Knowledge of Activity Recognition across Sensor Networks. Proc. Int. Conf. on Pervasive Computing (2010) pp Perkowitz et al. Mining models of human activities from the web. Proc. 13th Int. Conf. on World Wide Web (2004) pp Ward et al. Performance metrics for activity recognition. ACM Trans. on Intelligent Systems and Technology (2011) vol. 2 (1) Logan et al. A long-term evaluation of sensing modalities for activity recognition. Proc. Int. Conf. on Ubiquitous Computing (2007) pp Patterson et al. Fine-Grained Activity Recognition by Aggregating Abstract Object Usage. Proc. IEEE Int. Symp. on Wearable Computers (2005)