1. Using Expectations to Drive Cognitive Behavior Unmesh Kurup Christian Lebiere, Tony Stentz, Martial Hebert Carnegie Mellon University

2. Cognitive Decision Cycle t+1 Calculate Mismatch World High-level Cognition Retrieve Response World Action Prediction t-1t Cognition Cognition is driven by Expectations/Predictions.

3. Pedestrian Tracking & Behavior Classification Goals: Investigate use of expectations Integrate with perception Run both offline & real-time

4. Integrated System

5. Partial Matching & Blending Chunk2 isa location- chunk id person2 nextx 1010 nexty 500 Chunk3 isa location- chunk id person3 nextx 187 nexty 313 Chunk4 isa location- chunk id person1 nextx 299 nexty 100 +retrieval> isa location- chunk id person1 nextx 300 Chunk1 isa location- chunk id person1 nextx 255 nexty 100 Chunk4 isa location- chunk id person1 nextx 299 nexty 100 Declarative Memory Partial Matches Chunk5 isa location- chunk id person1 nextx 293.91 nexty 100 Blended result Chunk1 isa location- chunk id person2 nextx 255 nexty 100

6. Using Expectations: Tracking Chunk-type visual-location idX YDxDyNextxNexty Foreach Object o: +blending> isa visual-location id o compare to (x,y)s from perception pick thresholded closest match, calculate newdx, newdy, newx, newy +imaginal> isa visual-location id o …

7. Features straight1 straight2detourleftstraight3veer Features: BehaviorFeatures Normal – Straightstraight1, straight2, straight3 Normal – Leftstraight1, straight2, left Peekstraight1, detour, left, no-chk-pt BehaviorFeatures Detourstraight1, detour, straight3, chk-pt Veerstraight1, straight2, left, veer, chk-pt Walkbackstraight1, straight2, left, straight2, straight1, chk-pt

8. Using Expectations: Detecting Features from Data Straight & Left Deviation from expected location indicates a point of interest

9. Foreach location +blending> isa visual-location x =x y =y compare to (x,y)s from perception if path deviates more than threshold, mismatch! +imaginal> isa visual-location id o … Cluster points into regions

10. Detected Features

11. Data Combined Arms Collective Training Facility(CACTF) at Fort Indiantown Gap, PA. 4 examples. 3/1 split. Multiple behavior set – 10 behaviors.

12. Behaviors Straight & Left Peek Detour Veer Walkback

13. Results Hand-coded Model (Single Behavior Set) Hand-coded Model (Multiple Behavior Set) Made99.3%Made46.5% Correct99.15%Correct30.2% Incorrect0.15%Incorrect16.3% Learning Model (Single Behavior Set) Learning Model (Multiple Behavior Set) Made86.1%Made82.4% Correct68%Correct43.8% Incorrect18.1%Incorrect38.6%

14. Future Work – Semantic Labels

15. Future Work – Using Semantic Labels BehaviorFeatures (Spatial)Features (Semantic) Normal – Straightstraight1, straight2, straight3 Sidewalk, Pavement Normal – Leftstraight1, straight2, left Sidewalk Peekstraight1, detour, left, no-chk-pt Pavement, Sidewalk Detourstraight1, detour, straight3, chk-pt Pavement Veerstraight1, straight2, left, veer, chk-pt Sidewalk, Pavement Walkbackstraight1, straight2, left, straight2, straight1, chk- pt Sidewalk

16. Future Work Generic model of monitoring using expectations Learn expectations Monitor for deviations from expectations – Signal failure – Provide for recovery

17. Collaborators Max Bajracharya, JPL Bob Dean, GDRS Brad Stuart, GDRS FMS lab, CMU