Recognizing Human Activity from Sensor Data Henry Kautz University of Washington Computer Science & Engineering graduate students: Don Patterson, Lin Liao CSE faculty: Dieter Fox, Gaetano Borriello UW School of Medicine: Kurt Johnson Intel Research: Matthai Philipose, Tanzeem Choudhury

Converging Trends… Pervasive sensing infrastructure GPS enabled phones RFID tags on all consumer products Wireless motes Breakthroughs in core artificial intelligence After “AI boom” fizzled, basic science went on… Advances in algorithms for probabilistic reasoning and machine learning Bayesian networks Stochastic sampling Last decade: 10 variables  1,000,000 variables Healthcare crisis Epidemic of Alzheimer’s Disease Deinstitutionalization of the cognitively disabled Nationwide shortage of caretaking professionals

...An Opportunity Develop technology to Support independent living by people with cognitive disabilities At home At work Throughout the community Improve health care Long term monitoring of activities of daily living (ADL’s) Intervention before a health crisis

The University of Washington Assisted Cognition Project Synthesis of work in Ubiquitous computing Artificial intelligence Human-computer interaction ACCESS Support use of public transit CARE ADL monitoring and assistance

This Talk Building models of everyday plans and goals From sensor data By mining textual description By engineering commonsense knowledge Tracking and predicting a user’s behavior Noisy and incomplete sensor data Recognizing user errors First steps toward proactive assistive technology

ACCESS Assisted Cognition in Community, Employment, & Support Settings Supported by The National Institute on Disability & Rehabilitation Research (NIDDR) The National Science Foundation (NSF) Learning & Reasoning About Transportation Routines

Task Given a data stream from a wearable GPS unit... Infer the user’s location and mode of transportation (foot, car, bus, bike,...) Predict where user will go Detect novel behavior User errors? Opportunities for learning?

Why Inference Is Not Trivial People don’t have wheels Systematic GPS error We are not in the woods Dead and semi-dead zones Lots of multi-path propagation Inside of vehicles Inside of buildings Not just location tracking Mode, Prediction, Novelty

GPS Receivers We Used Nokia 6600 Java Cell Phone with Bluetooth GPS unit GeoStats wearable GPS logger

Geographic Information Systems Bus routes and bus stops Data source: Metro GIS Street map Data source: Census 2000 Tiger/line data

Architecture Learning Engine Inference Engine GIS Database Goals Paths Modes Errors

Probabilistic Reasoning Graphical model: Dynamic Bayesian network Inference engine: Rao-Blackwellised particle filters Learning engine: Expectation-Maximization (EM) algorithm

Graphical Model (Version 1) Transportation Mode Velocity Location Block Position along block At bus stop, parking lot,...? GPS Offset Error GPS signal

Rao-Blackwellised Particle Filtering Inference: estimate current state distribution given all past readings Particle filtering Evolve approximation to state distribution using samples (particles) Supports multi-modal distributions Supports discrete variables (e.g.: mode) Rao-Blackwellisation Each particle includes a Kalman filter to represent distribution over positions Improved accuracy with fewer particles

Tracking blue = foot green = bus red = car

Learning User model = DBN parameters Transitions between blocks Transitions between modes Learning: Monte-Carlo EM Unlabeled data 30 days of one user, logged at 2 second intervals (when outdoors) 3-fold cross validation

Results Model Mode Prediction Accuracy Decision Tree (supervised) 55% Prior w/o bus info60% Prior with bus info78% Learned84%

Probability of correctly predicting the future City Blocks Prediction Accuracy How can we improve predictive power?

Transportation Routines BA Goals work, home, friends, restaurant, doctor’s,... Trip segments Home to Bus stop A on Foot Bus stop A to Bus stop B on Bus Bus stop B to workplace on Foot Work “Learning & Inferring Transportation Routines”, Lin Liao, Dieter Fox, & Henry Kautz, AAAI-2004 Best Paper Award

Hierarchical Model Transportation mode x= GPS reading Goal Trip segment x k-1 z k-1 zkzk xkxk m k-1 mkmk t k-1 tktk g k-1 gkgk

Hierarchical Learning Learn flat model Infer goals Locations where user is often motionless Infer trip segment begin / end points Locations with high mode transition probability Infer trips segments High-probability single-mode block transition sequences between segment begin / end points Perform hierarchical EM learning

Inferring Goals

Inferring Trip Segments Going to workGoing home

Correct goal and route predicted 100 blocks away

Novelty & Error Detection Approach: model-selection Run several trackers in parallel Tracker 1: learned hierarchical model Tracker 2: untrained flat model Tracker 3: learned model with clamped final goal Estimate the likelihood of each tracker given the observations

Detect User Errors Untrained Trained Instantiated

Application: Opportunity Knocks Demonstration (by Don Patterson) at AAHA Future of Aging Services, Washington, DC, March, 2004

CARE Cognitive Assistance in Real-world Environments supported by the Intel Research Council Learning & Inferring Activities of Daily Living

Research Hypothesis Observation: activities of daily living involve the manipulation of many physical objects Cooking, cleaning, eating, personal hygiene, exercise, hobbies,... Hypothesis: can recognize activities from a time-sequence of object “touches” Such models are robust and easily learned or engineered

Sensing Object Manipulation RFID: Radio- frequency ID tags Small Semi-passive Durable Cheap

Where Can We Put Tags?

How Can We Sense Them? coming... wall-mounted “sparkle reader”

Example Data Stream

Making Tea

Building Models Core ADL’s amenable to classic knowledge engineering Open-ended, fine-grained models: infer from natural language texts? Perkowitz et al., “Mining Models of Human Activities from the Web”, WWW-2004

Experimental Setup Hand-built library of 14 ADL’s 17 test subjects Each asked to perform 12 of the ADL’s Data not segmented No training on individual test subjects

ActivityPrior WorkCARE Accuracy/Recall Personal Appearance 92/92 Oral Hygiene 70/78 Toileting 73/73 Washing up 100/33 Appliance Use 100/75 Use of Heating 84/78 Care of clothes and linen 100/73 Making a snack 100/78 Making a drink 75/60 Use of phone 64/64 Leisure Activity 100/79 Infant Care 100/58 Medication Taking 100/93 Housework 100/82 95/84 General Solution Quantitative Results Point Solution Quantitative Results Point Solution Anecdotal Results General Solution Anecdotal Results Pervasive Computing, Oct-Dec 2004

Current Directions Affective & physiological state agitated, calm, attentive,... hungry, tired, dizzy,... Interactions between people Human Social Dynamics Principled human-computer interaction Decision-theoretic control of interventions

Why Now? A goal of much work of AI in the 1970’s was to create programs that could understand the narrative of ordinary human experience This area pretty much disappeared Missing probabilistic tools Systems not able to experience world Lacked focus – “understand” to what end? Today: tools, grounding, motivation

Challenge to Nanotechnology Community Current sensors detect physical or physiological state: user mental state must be indirectly inferred To what can extend can nanotechnology afford direct access to a person’s emotions and intentions?