1. Integrating high- and low-level Expectations in Deliberative Agents Michele Piunti - michele.piunti@istc.cnr.itmichele.piunti@istc.cnr.it Institute of Cognitive Sciences and Technologies – ISTC, C.n.r. João Gonçalves - jcpgoncalves@sapo.ptjcpgoncalves@sapo.pt Instituto Superior Técnico - IST, Lisbon

2. 1.Towards an integrated architecture 1.Expectations, Emotions, Anticipation 2.High and Low level Expectations 3.From deliberative to anticipatory agents 2.Design 1.Mental States 2.Subjective Expected Utilities ISTC: Beliefs and Goal IST: Emotivectors 3.Experimental comparision and discussion 4.Future works Outline

3. Expectations, Emotions, Anticipation Agent anticipation in partially observable environments can rely in : The ability to adjust quickly to changes (making quick decisions with limited information and bounded resources) Catching world dynamics and regularities Building representations of future states (Expectations) Affective competences (Emotions) via Behavioral and Mental changes: –Long term : intention reconsideration, attention, resource allocation –Long term: appraisal, belief revision and learning

4. Explicit Vs. Implicit anticipatory representation: Expectations-enabled agents may or not make use of explicit representations of the future world state (and/or of the agent internal state). Agent architecture may compute, or not, explicit representation of these states Anticipatory representations and Computational Models Cognitive anticipatory agents can be endowed with expectations following different design approaches: Statistical learning, prediction mechanisms and component; Cognitive (and model driven) architectures: top down: Architecture for goal driven, affective and anticipatory agents bottom up: distribuited, drives, schema driven design (AKIRA), on line expectations beginning from perception and sensor motor control

5. Top down approach: from deliberative to anticipatory agents Deliberative (Goal directed) agents evaluate and chose between alternative courses of actions and their respective outcomes. Anticipatory competences require dealing with uncertainty and bounded knowledge about the future. We do not introduce Expectations as a new primitive of the architecture but in terms of agent epistemic states (Beliefs) and motivational states (Goals).

6. Scenario

7. From deliberative to anticipatory agents We introduce Expectations at many levels : 1.Weak, low level Expectations as moods and Mental States clustering attitudes in Reasoning. 2.Case based reasoning (means-end reasoning and action-selection processes) 3.Expectation ‘driven’ Deliberation: Subjective Expected Utility (as a function of the agent’s beliefs and desires (Bratmann88). IST: Emotivectors (Matinho 2005) Surprise: due to (and signal of) experienced mismatch between ’what is expected’ and ’what is perceived’ (at a given level of representation) Expectations are ’prerequisites’ for surprise. Affective States elicited by Surprise can be described in fuctional terms beginning from expectations

8. Clustering Mental States in Reasoning From the series of local observations of unexpected events stored in a short term memory, an agent controller periodically defines the mental state to adopt through a transition function. Expectations here have a weak (low level) representation (e.g. negative expectation of risk, threats)

9. Mental States: fuctional role Cautiousness elicits mental and behavioral changes on Short term: alert, to become more vigilant, to look ahead, to check better while and before moving (prudence against threats); Long term augmenting the control or doing the, action in another less risky way, using alternatives in repertoires. Excitement: increasing the explorative activity for searching for the ’good’ events. Lack of surprises produces a special mood: boredom. The persistence of boredom can bring to curiosity, whose outcome is to shift from exploitation to exploration attitudes.

10. Pay Offs Intention reconsideration is a costly process Space-Time payoffs Resources allocation To be cautious is advantegeous only in highly threatful environment (see: energy-time) Balance of resources is ‘environment dependent’

11. Subjective Expected Utility Foraging Task: drives and motivations to explore Location of Interest (LOI) are balanced: DriveToAloi = self_confidence.rule_LOI_a * reward_a.getAvg(); DriveToBloi = self_confidence.rule_LOI_b * reward_b.getAvg(); DriveToCloi = self_confidence.rule_LOI_b * reward_b.getAvg(); 1.Subjective Expected Utility: multiply subjective prevision to find valuables close to LOI and expected reward value (based on a k- history lenght items stored in a working memory). 2.Fully represented in domain of probability. 1.Meta-level planning: ε-Greedy strategies to select ‘best expected’ area (i.e. best SEU) to look for valuables at a. We integrated 2 mechanisms K-lenght history buffers ◄► emotivector predictors

12. Analysis of the ISTC Architecture Emotivectors model expectations –Prediction –Desired Value –Evaluation Expectations identified in 3 levels –Based on beliefs about the world –Associated with mental/emotional states –Associated with goal achievement

13. Expectations in 3 levels –Based on beliefs about the world –Associated with mental/emotional states –Associated with goal achievement

14. Modeling Beliefs with the Emotivector – Predictor Modeling Food Score –Scenario enhancement

15. Modeling Beliefs with the Emotivector – Affective Evaluation This feeling towards a certain kind of food reflects if its getting better or worst

16. Action Selection Previously the Agent used Subjective Expected Utility (SEU) –Just the predicted energy reward and probability of success Now it as a affective bias on the SEU for a specific kind of food –Affective Subjective Expected Utility

17. Preliminary Results Seasons No Seasons

18. Expectations in 3 levels –Based on beliefs about the world –Associated with mental/emotional states –Associated with goal achievement

19. Possible Integrations Associated with mental/emotional states –Emotivector monitors “rate/number” of positive events Anticipates mental state change –In part done by the weight decay mechanism Associated with goal achievement –Very context dependent

20. References [Castelfranchi et al., 2006a] C. Castelfranchi, R. Falcone, and M. Piunti. Agents with anticipatory behaviors: To be cautious in a risky environment. In Proc. of European Conf. on Artificial Intelligence, Trento, Italy., 2006. [Castelfranchi et al., 2006b] C. Castelfranchi, R. Falcone, and M. Piunti. Developing anticipatory and affective competences in MAS. In Proceedings of InternationalWorkshop on Multi-agent Systems and Simulation (ISC 2006), Palermo, Italy., 2006. [Castelfranchi, 2005] C. Castelfranchi. Mind as an anticipatory device: For a theory of expectations. pages 258–276, 2005. [C.Martino and Paiva, 2005] C.Martino and A. Paiva. Synthetic emotivectors. In In proc. of Social Intelligence and Interaction in Animals, Robots and Agents AISB, 2005. [C.Martino and Paiva, 2006] C.Martino and A. Paiva. Using anticipation to create believable behaviour. In In proc. Of the AAAI 2006 conference, 2006. [Falcone et al., 2007] R. Falcone, M. Piunti, and C. Castelfranchi. Surprise as shortcut for anticipation: clustering mental states in reasoning. In In Proc. of IJCAI-07 (to appear), Hyberadad, India., 2007.