Download presentation
Presentation is loading. Please wait.
1
Understanding Human Behavior from Sensor Data
Henry Kautz University of Washington
2
A Dream of AI Systems that can understand ordinary human experience
Work in KR, NLP, vision, IUI, planning… Plan recognition Behavior recognition Activity tracking Goals Intelligent user interfaces Step toward true AI
3
Plan Recognition, circa 1985
4
Behavior Recognition, circa 2005
5
Punch Line Resurgence of work in behavior understanding, fueled by
Advances in probabilistic inference Graphical models Scalable inference KR U Bayes Ubiquitous sensing devices RFID, GPS, motes, … Ground recognition in sensor data Healthcare applications Aging boomers – fastest growing demographic
6
This Talk Activity tracking from RFID tag data
ADL Monitoring Learning patterns of transportation use from GPS data Activity Compass Learning to label activities and places Life Capture Other recent research
7
This Talk Activity tracking from RFID tag data
ADL Monitoring Learning patterns of transportation use from GPS data Activity Compass Learning to label activities and places Life Capture Other recent research
8
Object-Based Activity Recognition
Activities of daily living involve the manipulation of many physical objects Kitchen: stove, pans, dishes, … Bathroom: toothbrush, shampoo, towel, … Bedroom: linen, dresser, clock, clothing, … We can recognize activities from a time-sequence of object touches
9
Image Courtesy Intel Research
Application ADL (Activity of Daily Living) monitoring for the disabled and/or elderly Changes in routine often precursor to illness, accidents Human monitoring intrusive & inaccurate Image Courtesy Intel Research
10
Sensing Object Manipulation
RFID: Radio-frequency identification tags Small No batteries Durable Cheap Easy to tag objects Near future: use products’ own tags
11
Wearable RFID Readers 13.56MHz reader, radio, power supply, antenna
12 inch range, hour lifetime Objects tagged on grasping surfaces
12
Experiment: ADL Form Filling
Tagged real home with 108 tags 14 subjects each performed 12 of 65 activities (14 ADLs) in arbitrary order Used glove-based reader Given trace, recreate activities
13
Results: Detecting ADLs
Activity Prior Work SHARP Personal Appearance 92/92 Oral Hygiene 70/78 Toileting 73/73 Washing up 100/33 Appliance Use 100/75 Use of Heating 84/78 Care of clothes and linen 100/73 Making a snack 100/78 Making a drink 75/60 Use of phone 64/64 Leisure Activity 100/79 Infant Care 100/58 Medication Taking 100/93 Housework 100/82 Legend Point solution General solution Inferring ADLs from Interactions with Objects Philipose, Fishkin, Perkowitz, Patterson, Hähnel, Fox, and Kautz IEEE Pervasive Computing, 4(3), 2004
14
Experiment: Morning Activities
10 days of data from the morning routine in an experimenter’s home 61 tagged objects 11 activities Often interleaved and interrupted Many shared objects Use bathroom Make coffee Set table Make oatmeal Make tea Eat breakfast Make eggs Use telephone Clear table Prepare OJ Take out trash
15
Baseline: Individual Hidden Markov Models
68% accuracy 11.8 errors per episode
16
Baseline: Single Hidden Markov Model
83% accuracy 9.4 errors per episode
17
Cause of Errors Observations were types of objects
Spoon, plate, fork … Typical errors: confusion between activities Using one object repeatedly Using different objects of same type Critical distinction in many ADL’s Eating versus setting table Dressing versus putting away laundry
18
Aggregate Features HMM with individual object observations fails
No generalization! Solution: add aggregation features Number of objects of each type used Requires history of current activity performance DBN encoding avoids explosion of HMM
19
Dynamic Bayes Net with Aggregate Features
88% accuracy 6.5 errors per episode
20
Improving Robustness DBN fails if novel objects are used
Solution: smooth parameters over abstraction hierarchy of object types
22
Abstraction Smoothing
Methodology: Train on 10 days data Test where one activity substitutes one object Change in error rate: Without smoothing: 26% increase With smoothing: 1% increase
23
Summary Activities of daily living can be robustly tracked using RFID data Simple, direct sensors can often replace (or augment) general machine vision Accurate probabilistic inference requires sequencing, aggregation, and abstraction Works for essentially all ADLs defined in healthcare literature D. Patterson, H. Kautz, & D. Fox, ISWC 2005 best paper award
24
This Talk Activity tracking from RFID tag data
ADL Monitoring Learning patterns of transportation use from GPS data Activity Compass Learning to label activities and places Life Capture Other recent research
25
Motivation: Community Access for the Cognitively Disabled
26
The Problem Using public transit cognitively challenging
Learning bus routes and numbers Transfers Recovering from mistakes Point to point shuttle service impractical Slow Expensive Current GPS units hard to use Require extensive user input Error-prone near buildings, inside buses No help with transfers, timing
27
Solution: Activity Compass
User carries smart cell phone System infers transportation mode GPS – position, velocity Mass transit information Over time system learns user model Important places Common transportation plans Mismatches = possible mistakes System provides proactive help
28
Technical Challenge Given a data stream from a GPS unit...
Infer the user’s mode of transportation, and places where the mode changes Foot, car, bus, bike, … Bus stops, parking lots, enter buildings, … Learn the user’s daily pattern of movement Predict the user’s future actions Detect user errors
29
Technical Approach Map is a directed graph G=(V,E) Location: Movement:
Edge e Distance d from start of edge Actual (displaced) GPS reading Movement: Mode { foot, car, bus } determines velocity range Change mode near bus stops & parking places Tracking (filtering): Given some prior estimate, Update position & mode according to motion model Correct according to next GPS reading So far, I have described the general picture of our system and some applications. In the rest of the presentation, I will focus on the technical approach of probabilistic reasoning. There are three key things in our system. For the .., we use…. Because of the time issue, I cannot discuss those techniques in details. So I will only give you a very rough picture for each of them and show the results of our experiments.
30
Dynamic Bayesian Network I
mk-1 mk Transportation mode xk-1 xk Edge, velocity, position qk-1 qk Data (edge) association zk-1 zk GPS reading Time k-1 Time k
31
Mode & Location Tracking
Measurements Projections Bus mode Car mode Foot mode Green Red Blue Explain why it switches from foot to bus, not car: because of the bus stops If people ask why no car particle at all (there are car particles, just their weights are too small that can not be displayed).
32
Learning Prior knowledge – general constraints on transportation use
Vehicle speed range Bus stops Learning – specialize model to particular user 30 days GPS readings Unlabeled data Learn edge transition parameters using expectation-maximization (EM)
33
How to improve predictive power?
Predictive Accuracy How to improve predictive power? Probability of correctly predicting the future City Blocks
34
Transportation Routines
Home A B Workplace Goal: intended destination Workplace, home, friends, restaurants, … Trip segments: <start, end, mode> Home to Bus stop A on Foot Bus stop A to Bus stop B on Bus Bus stop B to workplace on Foot Now the system could infer location and transportation modes, but wee still want to know more. We first show an example. Goal In order to get to the goal, the user may switch transportation mode. Thus we introduce the concept of trip segments.
35
Dynamic Bayesian Net II
gk-1 gk Goal tk-1 tk Trip segment mk-1 mk Transportation mode xk-1 xk Edge, velocity, position qk-1 qk Data (edge) association zk-1 zk GPS reading Time k-1 Time k
36
Unsupervised Hierarchical Learning
Use previous model to infer: Goals locations where user stays for a long time Transition points locations with high mode transition probability Trip segments paths connecting transition points or goals Learn transition probabilities Lower levels conditioned on higher levels Expectation-Maximization
37
Predict Goal and Path Predicted goal Predicted path
38
Improvement in Predictive Accuracy
39
Detecting User Errors Learned model represents typical correct behavior Model is a poor fit to user errors We can use this fact to detect errors! Cognitive Mode Normal: model functions as before Error: switch in prior (untrained) parameters for mode and edge transition After we learn the model, detect potential errors.
40
Dynamic Bayesian Net III
zk-1 zk xk-1 xk qk-1 qk Time k-1 Time k mk-1 mk tk-1 tk gk-1 gk ck-1 ck Cognitive mode { normal, error } Goal Trip segment Transportation mode Edge, velocity, position Data (edge) association GPS reading
41
Detecting User Errors In this episode, the person takes the bus home but missed the usual bus stop. The black disk is the goal and the cross mark is where the system think the person should get off the bus. The probability of normal and abnormal activities are represented using a bar here. The black part is the probability of being normal while the red part is the probability of being abnormal. At here, the person missed the bus stop and the probability of abnormal activity jump quickly.
42
Status Major funding by NIDRR Extension to indoor navigation
National Institute of Disability & Rehabilitation Research Partnership with UW Dept of Rehabilitation Medicine & Center for Technology and Disability Studies Extension to indoor navigation Hospitals, nursing homes, assisted care communities Wi-Fi localization Multi-modal interface studies Speech, graphics, text Adaptive guidance strategies
43
Papers Patterson, Liao, Fox, & Kautz, UBICOMP 2003
Inferring High Level Behavior from Low Level Sensors Patterson et al, UBICOMP-2004 Opportunity Knocks: a System to Provide Cognitive Assistance with Transportation Services Liao, Fox, & Kautz, AAAI 2004 (Best Paper) Learning and Inferring Transportation Routines
44
This Talk Activity tracking from RFID tag data
ADL Monitoring Learning patterns of transportation use from GPS data Activity Compass Learning to label activities and places Life Capture Other recent research
45
Task Learn to label a person’s Given Daily activities
working, visiting friends, traveling, … Significant places work place, friend’s house, usual bus stop, … Given Training set of labeled examples Wearable sensor data stream GPS, acceleration, ambient noise level, …
46
Applications Activity Compass Life Capture
Automatically assign names to places Life Capture Automated diary
47
Conditional Models HMMs and DBNs are generative models
Describe complete joint probability space For labeling tasks, conditional models are often simpler and more accurate Learn only P( label | observations ) Fewer parameters than corresponding generative model
48
Things to be Modeled Raw GPS reading (observed) Actual user location
Activities (time dependent) Significant places (time independent) Soft constraints between all of the above (learned)
49
Conditional Random Field
Undirected graphical model Feature functions defined on cliques Conditional probability proportional to exp( weighted sum of features ) Weights learned by maximizing (pseudo) likelihood of training data
50
Relational Markov Network
First-order version of conditional random field Features defined by feature templates All instances of a template have same weight Examples: Time of day an activity occurs Place an activity occurs Number of places labeled “Home” Distance between adjacent user locations Distance between GPS reading & nearest street
51
Global soft constraints
RMN Model s1 Global soft constraints p1 p2 Significant places a1 a2 a3 a4 a5 Activities g1 g2 g3 g4 g5 g6 g7 g8 g9 {GPS, location} time
52
Significant Places Previous work decoupled identifying significant places from rest of inference Simple temporal threshold Misses places with brief activities RMN model integrates Identifying significant place Labeling significant places Labeling activities
53
Efficient Inference Some features are expensive to handle by general inference algorithms E.g. belief propagation, MCMC Can dramatically speed up inference by associating inference procedures with feature templates Fast Fourier transform (FFT) to compute “counting” features O(n log2n) versus O(2n)
54
Results: Labeling One user, 1 week training data, 1 week testing data
Number of (new) significant places correctly labeled: 18 out of 19 Number of activities correctly labeled: 53 out of 61 Number of activities correctly labeled, if counting features not used: 44 out of 61
55
Results: Finding Significant Places
56
Results: Efficiency
57
Summary We can learn to label a user’s activities and meaningful locations using sensor data & state of the art relational statistical models (Liao, Fox, & Kautz, IJCAI 2005, NIPS 2006) Many avenues to explore: Transfer learning Finer grained activities Structured activities Social groups
58
This Talk Activity tracking from RFID tag data
ADL Monitoring Learning patterns of transportation use from GPS data Activity Compass Learning to label activities and places Life Capture Other recent research
59
Planning as Satisfiability
Unless P=NP, no polynomial time algorithm for SAT But: great practical progress in recent years 1980: 100 variable problems 2005: 100,000 variable problems Can we use SAT as a engine for planning? 1996 – competitive with state of the art ICAPS 2004 Planning Competition – 1st prize, optimal STRIPS planning Inspired research on bounded model-checking
60
Understanding Clause Learning
Modern SAT solvers extend Davis-Putnam backtrack search with clause learning Cache “reason” for each backtrack as an inferred clause Works in practice, but not clear why… Understanding clause learning using proof complexity (Sabharwal, Beame, & Kautz 2003, 2004, 2005) More powerful than tree-like resolution Separation from regular resolution Increase practical benefit by domain-specific variable- selection heuristics Linear-size proofs of pebbling graphs
61
Efficient Model Counting
SAT – can a formula be satisfied? #SAT – how many ways can a formula be satisfied? Compact translation discrete Bayesian networks #SAT Efficient model counting (Sang, Beame, & Kautz 2004, 2005) Formula caching Component analysis New branching heuristics Cachet – fastest model counting algorithm
62
Learning Control Policies for Satisfiability Solvers
High variance in runtime distributions of SAT solvers Over different random seeds for tie-breaking in branching heuristics Sometimes: infinite variance (heavy-tailed) Restarts can yield super-linear speedup Can learn optimal restart policies using features of instance + solver trace (Ruan, Kautz, & Horvitz 2001, 2002, 2003)
63
Other Projects Continuous time Bayesian networks
(Gopalratnam, Weld, & Kautz, AAAI 2005) Building social network models from sensor data Probabilistic speech act theory Modal Markov Logic
64
Conclusion: Why Now? An early goal of AI was to create programs that could understand ordinary human experience This goal proved elusive Missing probabilistic tools Systems not grounded in real world Lacked compelling purpose Today we have the mathematical tools, the sensors, and the motivation
65
Credits Graduate students: Colleagues: Funders:
Don Patterson, Lin Liao, Ashish Sabharwal, Yongshao Ruan, Tian Sang, Harlan Hile, Alan Liu, Bill Pentney, Brian Ferris Colleagues: Dieter Fox, Gaetano Borriello, Dan Weld, Matthai Philipose Funders: NIDRR, Intel, NSF, DARPA
Similar presentations
