1. Affective Computing: Machines with Emotional Intelligence Hyung-il Ahn MIT Media Laboratory

2. …doesn’t notice you are annoyed. [Doesn’t recognize your emotion] You express more annoyance. He ignores it. [Stupid about handling your emotion] He winks, and does a happy little dance before exiting. [Stupid about expressing emotion.]

3. Expressing emotions Recognizing emotions Handling another’s emotions Regulating emotions \ Utilizing emotions / (Salovey and Mayer 90, Goleman 95) Skills of Emotional Intelligence: if “have emotion”

4. Research Areas Robotic Computer - Recognizing another’s emotions - Expressing emotions - Handling another’s emotions Affective and Cognitive Decision Making - Regulating and utilizing emotions - Affect as a self-adapting control system Affect changes the operating characteristics of other three domains (cognition, motivation, behavior)

5. Recognizing Emotions

6. Recognition of three “basic” states:

7. Future “teacher for every learner”

8. Can we teach a chair to recognize behaviors indicative of interest and boredom? (Mota and Picard) Sit uprightLean ForwardSlump BackSide Lean

9. Boredom Interest

10. What can the sensor chair contribute toward inferring the student’s state: Bored vs. interested? Results (on children not in training data, Mota and Picard, 2003): 9-state Posture Recognition: 89-97% accurate High Interest, Low interest, Taking a Break: 69-83% accurate

11. Detecting, tracking, and recognizing facial expressions from video (IBM BlueEyes camera with MIT algorithms)

12. Autism Spectrum Conditions Center for Disease Control and Prevention (2005) –1 child in 166 has ASC

13. Mind-Read > Act > Persuade hmm … Roz looks busy. Its probably not a good time to bring this up Analysis of nonverbal cues Inference and reasoning about mental states Modify one’s actions Persuade others

14. Real time Mental State Inference Feature point tracking* Head pose estimation Facial feature extraction Head & facial action unit recognition Head & facial display recognition Mental state inference hmm … Let me think about this El Kaliouby and Robinson (2005) * Nevenvision face-tracker

15. Affective-Cognitive Mental States Complex Mental States (subset) Concentrating Disagreeing Interested Thinking Unsure Absorbed Concentrating Vigilant Disapproving Discouraging Disinclined Asking Curious Impressed Interested Brooding Choosing Thinking Thoughtful Baffled Confused Undecided Unsure Agreeing Assertive Committed Persuaded Sure Baron-Cohen et al. AUTISM RESEARCH CENTRE, CAMBRIDGE

16. Physically animated Robotic Computer (joint with Prof. Cynthia Breazeal) Goal: increase user movement without distraction and annoyance, further social-rapport building

17. Robotic Computer (RoCo): A physically animated computer Learning: the user can guide RoCo’s behavior by explicit and implicit rewards and punishments (Reinforcement Learning)

18. RoCo’s postures congruous to the user affect N=(17) “Stoop to Conquer” : Posture and affect interact to influence computer users’ comfort and persistence in problem solving tasks People tend to be more persistent and feel more comfortable when RoCo’s posture is congruous to their affective state

19. Procedure and Tasks Tracing Task: a solvable and an unsolvable puzzle Decision-making Task (in Experiment 2): to make subjects keep the target posture longer

20. Affective Cognitive Decision Making

21. (Example 1) Two-armed bandit gambling tasks The left arm has ‘Negative Valence’ Arousal (uncertainty) as ‘feeling uneasy’ Inspired by Bechara & Damasio’s IOWA gambling tasks (Bechara et al. 1997) The right arm has ‘Positive Valence’ Arousal (uncertainty) as ‘feeling lucky’

22. (Example 2) Decision making under risk Loss aversion: People strongly prefer avoiding losses than acquiring gains $3000 (Pr=1) $4000 (Pr=0.8) $ 0 (Pr=0.2) Expected value = $4000 * 0.8 + $0 * 0.2 = $3200 (Gain) > Expected value = $3000 (Gain) < ‘Risk-Seeking’ choices in the domain of ‘Likely Losses’ - $3000 (Pr=1) - $4000 (Pr=0.8) $ 0 (Pr=0.2) Expected value = - $4000 * 0.8 + $0 * 0.2 = - $3200 (Loss) < Expected value = - $3000 (Loss) > ‘Risk-Averse’ choices in the domain of ‘Likely Gains’ Option 1 Option 2

23. The PT (Prospect Theory) value function - Diminishing sensitivity: less sensitive to outliers for both gains and losses - Loss aversion: the function is steeper in the negative (loss) domain (Tversky & Kahneman) - Reference Dependence: gains and losses are defined relative to the reference point - Concave above the reference point - Convex below the reference point

24. Endowment Effect people place a higher value on objects they own relative to objects they do not. In one experiment, people demanded a higher price for a coffee mug that had been given to them but put a lower price on one they did not yet own. The endowment effect was described as inconsistent with standard economic theory which asserts that a person's willingness to pay (WTP) for a good should be equal to their willingness to accept (WTA) compensation to be deprived of the good. This hypothesis underlies consumer theory and indifference curves.economic willingness to pay willingness to acceptconsumer theory indifference curves The effect is related to loss aversion and status quo bias in prospect theory.loss aversionstatus quo bias prospect theory

25. (Example 3) Effects of mood on decision making Optimistic about judgments of future events Happiness Sadness Anger Fear Pessimistic judgments of future events, Risk-Aversive choices Optimistic judgments of future events, Risk-Seeking choices Reverse Endowment Effect (Lerner & Keltner 2000, 2001, 2004)

26. Subjective Value Function (mood influences decision making)

27. Affective Cognitive Learning and Decision Making A new computational framework for learning and decision making inspired by the neural basis of motivations and the role of emotions in human behaviors A motivational value (reward)-based learning theory: decision value = extrinsic (cognitive) value + intrinsic (affective) value extrinsic value from the cognitive (deliberative and analytic) systems intrinsic value from multiple affective systems such as Seeking, Fear, Rage, and other circuits. Probabilistic models: Cognition (cognitive state transition), Multiple affect circuits (Seeking, Joy, Anger, Fear, …), and Decision making model Any prior and learned knowledge can be incorporated for expecting the consequences of decisions (or computing the cognitive value)

28. Pr = 0.5 Success (r = 100) Fail (r = 0) To destroy the ring in Mordor with less effort Choice 1 Effort (r = -80) Choice 2 Effort (r = -30) Fear Anticipatory Emotions from Other Circuits Expected Values Cognitive Expectations choice 1 = 20, choice 2 = 20 Fearless/ Neutral / Fearful Mood Incidental Emotions 070-30 Reward Prob 20 Valenced Uncertainty Values Anticipatory Emotions from the Seeking Circuit choice 1 = positive, choice 2 = negative