1. DEPARTMENT of COMPUTER SCIENCE University of Rochester  Activities  Abductive Inference of Multi-Agent Interaction  Capture the Flag Data Collection  Representing Beliefs & Goals of Multiple Agents  Modal Markov Logic  Recognizing Indoor Activities using Multi-Modal Data  Fusing RFID and Machine Vision

2. DEPARTMENT of COMPUTER SCIENCE Multi-Agent Interaction  Many agent behaviors can only be understood in the context of the actions of other agents  Exercising?  Being chased?  Chasing someone?  Location alone provides a surprisingly rich source of information about behavior  GPS data can be used to learn a probabilistic model of a individual’s common activities (Liao, Fox, & Kautz 2007)  Goal: learn models of groups and interactive activities  Relational learning problem: ideal for ML  Needed: dataset of competitive & cooperative interaction

3. DEPARTMENT of COMPUTER SCIENCE Capture the Flag  Capture the Flag Data Collection  UR campus  Up to 150 x 300 m area  Complex topology  14 players, 8 games  GPS loggers  Accuracy varies 1-9 m  Average game 12 m

4. DEPARTMENT of COMPUTER SCIENCE Start of Game

5. DEPARTMENT of COMPUTER SCIENCE End of Game 1.red & orange guarded by green 2.green leaves prisoners 3.violet releases red & orange 4.red captures flag

6. DEPARTMENT of COMPUTER SCIENCE Ground Truth  For supervised learning methods, need to create a labeled training set  First attempt: record voice annotations from players  Failed: players too involved to accurately comment on their actions  Second attempt: post-hoc annotation  Created general annotation tool for relations over GPS streams

7. DEPARTMENT of COMPUTER SCIENCE Supervised Weight Learning  Discrete features calculated from GPS streams  Supervised learning applied to simple 2-slice model  Precision: 46% (second by second)  Recall 64%  12 hours to label 1 hour of training data

8. DEPARTMENT of COMPUTER SCIENCE Observations  Humans can accurately perceive interactive behaviors  High agreement between annotators  GPS noise often obscures geometric details  Reasoning about intention over extended temporal context disambiguates action

9. DEPARTMENT of COMPUTER SCIENCE Year 2 Goals  Improve quality of data (features) using physical constraints  Hard constraints: walls  Soft constraints: paths  CRF “snapping” tool  Model long temporal dependencies  Unsupervised learning: discover behaviors, tactics, strategies

10. DEPARTMENT of COMPUTER SCIENCE Representing Beliefs & Goals of Multiple Agents  Abduction often requires reasoning about the establishment of “propositional attitudes”  Belief, desire, intention, commitment, …  Example: principles of communication:  If A tells B that P, then A believes P.  If A tells B that P, then B will believe that A wants B to believe P.  If A is cooperative with B, and B wants P, then A will want P.  Such principles are defeasible

11. DEPARTMENT of COMPUTER SCIENCE Modal Operators  In logic  Predicates relate one object to another  Modal operators relate objects (agents) to propositions (sentences)  Different modalities can be axiomatically characterized  Deductive closure:  Transitivity:

12. DEPARTMENT of COMPUTER SCIENCE Modal Operators in Markov Logic  ML defines a probability distribution over propositional truth assignments  Idea: define probability distribution over assignments that are modally consistent Non-modal atoms Modal atoms Modal consistency check

13. DEPARTMENT of COMPUTER SCIENCE Inference  Complexity of consistency check  Depends on target modal logic  Belief (KD45):  Unbounded nesting: PSPACE-complete  Bounded nesting: NP-complete  Modal Markov Logic Inference  Rejection sampling  Optimizations  Cache g(M)  Compute g(M) incrementally

14. DEPARTMENT of COMPUTER SCIENCE Year 2 Goals  Implement MML in Alchemy  Applications  Understanding indirect speech acts  Capture the flag  Establishing knowledge by perception  Representing degrees of belief  Functional modal operators