1. Robust Activity Recognition Henry Kautz University of Washington Computer Science & Engineering graduate students: Don Patterson, Lin Liao, Krzysztof Gajos, Karthik Gopalratnam CSE faculty: Dieter Fox, Gaetano Borriello UW School of Medicine: Kurt Johnson, Pat Brown, Brian Dudgeon, Mark Harniss Intel Research: Matthai Philipose, Mike Perkowitz, Ken Fishkin, Tanzeem Choudhury

2. In the Not Too Distant Future... Pervasive sensing infrastructure GPS enabled phones RFID tags on all consumer products Electronic diaries (MS SenseCam) Healthcare crisis Aging baby boomers – epidemic of Alzheimer’s Disease Deinstitutionalization of the cognitively disabled Nationwide shortage of caretaking professionals

3. ...An Opportunity Develop technology to Support independent living by people with cognitive disabilities At home At work Throughout the community Improve health care Long term monitoring of activities of daily living (ADL’s) Intervention before a health crisis

4. The UW Assisted Cognition Project Synthesis of work in Ubiquitous computing Artificial intelligence Human-computer interaction ACCESS Support use of public transit UW CSE & Rehabilitation Medicine CARE ADL monitoring and assistance UW CSE & Intel Research

5. This Talk Building models of everyday plans and goals From sensor data By mining textual description By engineering commonsense knowledge Tracking and predicting a user’s behavior Noisy and incomplete sensor data Recognizing user errors First steps

6. ACCESS Assisted Cognition in Community, Employment, & Support Settings Supported by the National Institute on Disability & Rehabilitation Research (NIDDR) Learning & Reasoning About Transportation Routines

7. Task Given a data stream from a wearable GPS unit... Infer the user’s location and mode of transportation (foot, car, bus, bike,...) Predict where user will go Detect novel behavior User errors? Opportunities for learning?

8. Why Inference Is Not Trivial People don’t have wheels Systematic GPS error We are not in the woods Dead and semi-dead zones Lots of multi-path propagation Inside of vehicles Inside of buildings Not just location tracking Mode, Prediction, Novelty

9. GPS Receivers We Used Nokia 6600 Java Cell Phone with Bluetooth GPS unit GeoStats wearable GPS logger

10. Geographic Information Systems Bus routes and bus stops Data source: Metro GIS Street map Data source: Census 2000 Tiger/line data

11. Architecture Learning Engine Inference Engine GIS Database Goals Paths Modes Errors

12. Probabilistic Reasoning Graphical model: Dynamic Bayesian network Inference engine: Rao-Blackwellised particle filters Learning engine: Expectation-Maximization (EM) algorithm

13. Flat Model: State Space Transportation Mode Velocity Location Block Position along block At bus stop, parking lot,...? GPS Offset Error GPS signal

16. Motion Model for Mode of Transportation

17. Rao-Blackwellised Particle Filtering Inference: estimate current state distribution given all past readings Particle filtering Evolve approximation to state distribution using samples (particles) Supports multi-modal distributions Supports discrete variables (e.g.: mode) Rao-Blackwellisation Particles include distributions over variables, not just single samples Improved accuracy with fewer particles

18. Tracking blue = foot, green = bus, red = car

19. Learning User model = DBN parameters Transitions between blocks Transitions between modes Learning: Monte-Carlo EM Unlabeled data 30 days of one user, logged at 2 second intervals (when outdoors) 3-fold cross validation

20. Results Model Mode Prediction Accuracy Decision Tree (supervised) 55% Prior w/o bus info60% Prior with bus info78% Learned84%

21. Probability of correctly predicting the future City Blocks Prediction Accuracy How can we improve predictive power?

22. Transportation Routines BA Goals work, home, friends, restaurant, doctor’s,... Trip segments Home to Bus stop A on Foot Bus stop A to Bus stop B on Bus Bus stop B to workplace on Foot Work

23. Hierarchical Model x k-1 z k-1 zkzk xkxk m k-1 mkmk Transportation mode x= GPS reading t k-1 tktk g k-1 gkgk Goal Trip segment

24. Hierarchical Learning Learn flat model Infer goals Locations where user is often motionless Infer trip segment begin / end points Locations with high mode transition probability Infer trips segments High-probability single-mode block transition sequences between segment begin / end points Perform hierarchical EM learning

25. Inferring Goals

26. Inferring Trip Segments Going to workGoing home

27. Correct goal and route predicted 100 blocks away

28. Application: Opportunity Knocks Demonstrated at AAHA Future of Aging Services, Washington, DC, March, 2004

29. Novelty Detection Approach: model-selection Run two trackers in parallel Tracker 1: learned hierarchical model Tracker 2: untrained flat model Estimate the likelihood of each tracker given the observations

30. Missing the bus stop

31. Novelty Detection

33. CARE Cognitive Assistance in Real-world Environments supported by the Intel Research Council Learning & Inferring Activities of Daily Living

34. Research Hypothesis Observation: activities of daily living involve the manipulation of many physical objects Cooking, cleaning, eating, personal hygiene, exercise, hobbies,... Hypothesis: can recognize activities from a time-sequence of object “touches” Such models are robust and easily learned or engineered

35. Sensing Object Manipulation RFID: Radio- frequency ID tags Small Semi-passive Durable Cheap

36. Where Can We Put Tags?

37. How Can We Sense Them? coming... wall-mounted “sparkle reader”

38. Example Data Stream

39. Technical Approach Define (or learn) activities in simple, high-level language Multi-step, partially-ordered activities Varying durations Probabilistic association between activities and objects Compile to a DBN Infer behavior using particle filtering

40. Making Tea

41. Activity Library

42. Building Models Core ADL’s amenable to classic knowledge engineering Open-ended, fine-grained models: infer from natural language texts? Perkowitz et al., “Mining Models of Human Activities from the Web”, WWW-2004

43. Translation to DBN Tricky issues: Time Partial orders Object-use probabilities 80% chance of using the teapot sometime during the “heat water” step Instantaneous probability of seeing teapot is not fixed! Consider: 100% chance of using teapot if making tea

44. DBN Encoding: Duration DtDt AtAt A t+1 D t+1

45. DBN Encoding: Partial Orders PtPt AtAt A t+1 PtPt

46. DBN Encoding: Object Probabilities ztzt DtDt AtAt OtOt HtHt Instantaneous probability of touching an object cannot be a constant

47. DBN Encoding ztzt PtPt DtDt AtAt OtOt HtHt A t+1 D t+1 H t+1 PtPt

48. What’s in a Particle? Sample of Activity Starting time – sufficient to represent distribution of Duration History list of objects Partial-order “credits”

49. Experimental Setup Hand-built library of 14 ADL’s 17 test subjects Each asked to perform 12 of the ADL’s Data not segmented No training on individual test subjects

50. Sample Output

51. Results ADLPrecisionRecall 1Grooming92 2Tooth brushing7078 3Toileting73 4Dishwashing10033 5Housecleaning10075 6Appliance use8478 7Adjust furnace10073 8Laundry10078 9Prepare snack7560 10Prepare beverage64 11Use telephone10079 12Leisure activities10058 13Infant care10093 14Take medication10082 Overall8873

52. Key Next Steps Parameter learning Timing Object probabilities Structure learning New activities from sensor data Efficient inference for Interrupted activities Abandoned activities Malformed activities Relational models Hierarchical classes of objects Hierarchical classes of activities

53. Ultimately... Affective state agitated, calm, attentive,... Physiological states hungry, tired, dizzy,... Interactions between people T. Choudhury – Social dynamics Principled human-computer interaction Decision-theoretic control of interventions

54. Why Now? A goal of much work of AI in the 1970’s was to create programs that could understand the narrative of ordinary human experience This area pretty much disappeared Missing probabilistic tools Systems not able to experience world Lacked focus – “understand” to what end? Today: the tools, the sensors, motivation

55. That Other Talk... Combining Component Caching and Clause Learning for Effective Model Counting Beame, Bacchus, Kautz, Pitassi, & Sang (SAT 2004, Vancouver BC) Unifies algorithms for SAT and Bayesian inference DPLL-based, generalizes recursive conditioning Exact inference in large, non-tree-like networks Need to solve #P? Let me know!