The New Age of Commonsense Reasoning Henry Kautz University of Rochester

The Idea of Commonsense Reasoning Programs with Common Sense (McCarthy 1959): – Use declarative representations of knowledge and general reasoning methods to Understand the physical world Understand human behavior Motivation – Tasks: “advice taker”, story understanding – Step towards true AI

Commonsense Reasoning, circa 1985 Declarative and general Missing: grounding, learning, probabilities, scalable inference

Commonsense Reasoning, circa 2008

Methods for grounding theories in sensor data Graphical models for learning & probabilistic inference Practical (low-exponent) reasoning algorithms

New Applications Caregiving systems – AI-based assistive technology – Assisted cognition Automated planning – Supra-expert level Verification Natural language processing...

This Talk Monitoring activities of daily living Learning and guiding users through transportation plans Planning as satisfiability

Object-Based Activity Recognition Activities of daily living involve the manipulation of many physical objects – Kitchen: stove, pans, dishes, … – Bathroom: toothbrush, shampoo, towel, … – Bedroom: linen, dresser, clock, clothing, … We can recognize activities from a time- sequence of object touches (Philipose et al. 2004)

Applications ADL (Activity of Daily Living) monitoring for the disabled and/or elderly – Changes in routine often precursor to illness, accidents – Human monitoring intrusive & inaccurate Image Courtesy Intel Research

Sensing Object Manipulation RFID: Radio- frequency identification tags – Small – Durable – Cheap Easy to tag objects – Near future: use products’ own tags IRS bracelet reader

Experiment: Morning Activities 10 days of data from the morning routine in an experimenter’s home (Patterson, Fox, & Kautz 2005) 11 activities, 61 tagged objects – Often interleaved and interrupted – Many shared objects Use bathroomMake coffeeSet table Make oatmealMake teaEat breakfast Make eggsUse telephoneClear table Prepare OJTake out trash

Most Likely Single-State HMM Each activity modeled as an independent single- state Hidden Markov Model Captures relationship between activities and objects Compute probabilities from labeled training data Compute most likely model at each time point 68% accuracy

HMM with One State per Activity Single HMM where each activity is a state Captures probability of transitioning between activities Compute most state sequence 88% accuracy

HMM with Multiple States per Activity HMM with (# objects) states per activity Idea: model internal structure of activities as well as transitions between activies 87% accuracy – Performance degrades

Model Complexity Tradeoff Too few parameters: cannot fit data Too many parameters: poor generalization Leta = # different activities b = # objects per activity Number of parameters: – HMM with 1 state per activity: O(a 2 + ab) – Too simple – HMM with b states per activity: O(a 2 b 2 ) – Too complex

Interleaved HMM Idea: explicitly distinguish progress within an activity from switching between activities (Bai & Kautz 2008) Each activity modeled by an individual HMM User modeled by a set of activities – One activity is active, others are suspended – When an observation is made, credit it to the current activity, or switch to a different activity Frontier

Interleaved HMM Filtering

Results Model# ParametersAccuracy Most likely single-state HMM ab68% HMM with one state per activity a 2 + ab88% HMM with (# objects) states per activity a2b2a2b2 87% Interleaved HMM with (# objects) states per activity a 2 +ab 2 98%

Interleaved HMM Confusion Matrix

Current Work: From Activity Recognition to ADL Prompting Collaboration with Attention Control Systems to integrate activity monitoring into PEAT (Planning and Execution Assistant and Trainer) – DARPA Small Business Innovative Research

This Talk Monitoring activities of daily living Learning and guiding users through transportation plans Planning as satisfiability

Motivation: Community Access for Persons with Cognitive Disabilities

Problems Using public transit cognitively challenging – Learning bus routes and transfers – Recovering from mistakes Point to point shuttle service impractical – Slow – Expensive Current GPS units hard to use – Require extensive user input – No help with transfers, timing

Solution: Opportunity Knocks User carries GPS-enabled cell phone System infers transportation use System learns model of typical user behavior Novel events = possible user errors

Modeling Challenge Create a model of a user’s – Significant places – Modes of transportation – Transportation plans Approach: – Encode a commonsense theory of transportation use as a Dynamic Bayesian Network (DBN) – Learn the parameters of the DBN from raw GPS data using Expectation-Maximization

Transportation Plans BA Goal: ultimate destination ̶ Home, work, friends, stores, doctors, … Places: goals or intermediate waypoints Trip segments: Plan = sequence of trip segments ̶ Home to Bus stop A on Foot ̶ Bus stop A to Bus stop B on Bus ̶ Bus stop B to workplace on Foot WorkHome

Detecting User Errors Learned model represents typical correct behavior – Model is a poor fit to user errors We can use this fact to detect errors! Novelty – Normal: model functions as before – Abnormal: switch in prior (untrained) parameters for mode and edge transition

GPS reading Edge, velocity, position GPS / pathway association Transportation mode Trip segment Goal Novelty Dynamic Bayesian Net z k-1 zkzk x k-1 xkxk  k-1 kk Time k-1 Time k m k-1 mkmk t k-1 tktk g k-1 gkgk c k-1 ckck

Predict Goal and Path

Detecting Novel Events

Current Work Modeling multi-person interactions from GPS data – Chasing, meeting, hiding, … – Capture the flag game – Strategic understanding Generating prompts using a Markov Decision Process model

This Talk Monitoring activities of daily living Learning and guiding users through transportation plans Planning as satisfiability

Reasoning about the Physical World The planning problem: – Given representations of An initial state of affairs A desired (goal) state of affairs A set of actions defined by preconditions and effects – Find a (shortest) sequence of actions that transforms the initial state into a goal state All useful formalizations of planning are NP- complete or worse – In practice: desire low-exponential algorithms

Planning as Inference Advice taker (McCarthy 1959) – Proof checker Planning as first-order theorem proving (Green 1969) – Computationally infeasible STRIPS (Fikes & Nilsson 1971) – Limited theorem proving + state-space search – Very hard UCPOP (Weld 1992) – Specialized theorem prover for partial-order planning – Starting to scale up SATPLAN (Kautz & Selman 1996, 1999, 2000, 2006, 2007) – Planning as general propositional reasoning – Solves many hard problems (60+ steps) optimally

Planning as Satisfiability Time = bounded sequence of integers Translate planning problem to a Boolean formula Find a satisfying truth assignment using a SAT engine Translate truth assignment to a plan

Benchmark Results International Planning Competitions (2004 & 2006) – 1 st place for deterministic optimal planning

Key Improvements Stochastic local search Clause learning Restart strategies

Future Work Extend SATPLAN approach to – Planning with metric costs & resources – Planning under uncertainty – Integrated planning & plan recognition

Supra-Human Planning International planning competition benchmarks ; Time ; ParsingTime 0.00 ; MakeSpan 8 0: (TURN_TO SATELLITE0 STAR2 STAR8) [1] 0: (TURN_TO SATELLITE1 STAR0 GROUNDSTATION3) [1] 0: (TURN_TO SATELLITE2 STAR2 STAR4) [1] 0: (TURN_TO SATELLITE3 STAR2 PHENOMENON9) [1] 0: (TURN_TO SATELLITE4 STAR2 PHENOMENON9) [1] 0: (SWITCH_ON INSTRUMENT0 SATELLITE0) [1] 0: (SWITCH_ON INSTRUMENT3 SATELLITE1) [1] 0: (SWITCH_ON INSTRUMENT4 SATELLITE2) [1] 0: (SWITCH_ON INSTRUMENT7 SATELLITE3) [1] 0: (SWITCH_ON INSTRUMENT8 SATELLITE4) [1] 1: (CALIBRATE SATELLITE1 INSTRUMENT3 STAR0) [1] 1: (CALIBRATE SATELLITE2 INSTRUMENT4 STAR2) [1] 1: (CALIBRATE SATELLITE3 INSTRUMENT7 STAR2) [1] 1: (CALIBRATE SATELLITE4 INSTRUMENT8 STAR2) [1] 1: (TURN_TO SATELLITE0 STAR0 STAR2) [1] 2: (TURN_TO SATELLITE1 STAR10 STAR0) [1] 2: (TURN_TO SATELLITE2 PHENOMENON7 STAR2) [1] 2: (TURN_TO SATELLITE3 STAR5 STAR2) [1] 2: (TURN_TO SATELLITE4 PLANET6 STAR2) [1] 2: (CALIBRATE SATELLITE0 INSTRUMENT0 STAR0) [1] 3: (TURN_TO SATELLITE0 PLANET6 STAR0) [1] 3: (TAKE_IMAGE SATELLITE1 STAR10 INSTRUMENT3 THERMOGRAPH2) [1] 3: (TAKE_IMAGE SATELLITE2 PHENOMENON7 INSTRUMENT4 INFRARED1) [1] 3: (TAKE_IMAGE SATELLITE2 PHENOMENON7 INSTRUMENT4 IMAGE3) [1] 3: (TAKE_IMAGE SATELLITE2 PHENOMENON7 INSTRUMENT4 INFRARED0) [1] 3: (TAKE_IMAGE SATELLITE3 STAR5 INSTRUMENT7 IMAGE3) [1] 3: (TAKE_IMAGE SATELLITE4 PLANET6 INSTRUMENT8 INFRARED0) [1] 3: (TAKE_IMAGE SATELLITE4 PLANET6 INSTRUMENT8 INFRARED1) [1] 4: (TURN_TO SATELLITE2 PHENOMENON15 PHENOMENON7) [1] 4: (TURN_TO SATELLITE3 STAR8 STAR5) [1] 4: (TURN_TO SATELLITE1 STAR1 STAR10) [1] 4: (TURN_TO SATELLITE4 GROUNDSTATION3 PLANET6) [1] 5: (TURN_TO SATELLITE0 PLANET13 PLANET6) [1] 5: (TURN_TO SATELLITE1 PLANET14 STAR1) [1] 5: (TURN_TO SATELLITE4 PHENOMENON7 GROUNDSTATION3) [1] 5: (TAKE_IMAGE SATELLITE2 PHENOMENON15 INSTRUMENT4 INFRARED1) [1] 5: (TAKE_IMAGE SATELLITE2 PHENOMENON15 INSTRUMENT4 INFRARED0) [1] 5: (TAKE_IMAGE SATELLITE3 STAR8 INSTRUMENT7 IMAGE3) [1] 6: (TURN_TO SATELLITE2 STAR17 PHENOMENON15) [1] 6: (TURN_TO SATELLITE3 PLANET16 STAR8) [1] 6: (TURN_TO SATELLITE4 STAR11 PHENOMENON7) [1] 6: (TAKE_IMAGE SATELLITE0 PLANET13 INSTRUMENT0 SPECTROGRAPH4) [1] 6: (TAKE_IMAGE SATELLITE1 PLANET14 INSTRUMENT3 THERMOGRAPH2) [1] 7: (TAKE_IMAGE SATELLITE2 STAR17 INSTRUMENT4 INFRARED0) [1] 7: (TAKE_IMAGE SATELLITE3 PLANET16 INSTRUMENT7 IMAGE3) [1] 7: (TAKE_IMAGE SATELLITE4 STAR11 INSTRUMENT8 INFRARED1) [1] 7: (TURN_TO SATELLITE0 PHENOMENON9 PLANET13) [1] 7: (TURN_TO SATELLITE1 STAR4 PLANET14) [1]

Supra-Human Planning Industrial planning – Lithographic printing production planning – Complex set of discrete & geometric constraints – Expert+ performance (40 job ganging) – Domain-specific implementation of SAT techniques

But, also… Discovery of social relationships in consumer photo collections Recognition of activities of daily living using multi-scale, multi- perspective vision and RFID Applying SAT techniques to model-counting and probabilistic inference

Conclusion An early goal of AI was to create programs that exhibited commonsense This goal proved elusive – Missing efficient methods for logical & probabilistic reasoning – Lacked grounding in real world – Lacked compelling applications Today we have the tools, the sensors, and the motivation

Acknowledgements Tongxin Bai, Jeff Bilmes, Gaetano Borriello, Tanzeem Choudhury, Kate Deibel, Oren Etzioni, Dieter Fox, Carla Gomes, Craig Harman, Harlan Hile, Eric Horvitz, Kurt Johnson, Lin Liao, Alan Liu, Joseph Modayil, Don Patterson, Parag, Sangho Park, Bill Pentney, Matthai Philipose, Yongshao Ruan, Ashish Sabharwal, Tian Sang, Bart Selman, Dan Weld, Danny Wyatt