Affective Facial Expressions Facilitate Robot Learning Joost Broekens Pascal Haazebroek LIACS, Leiden University, The Netherlands.

Slides:



Advertisements
Similar presentations
Chapter 9 Motivation Explain what motivation is and why managers need to be concerned about it Describe from the perspectives of expectancy theory and.
Advertisements

Computational Aspects of Emotion in Adaptive Behavior Joost Broekens, Walter Kosters, Fons Verbeek LIACS, Leiden University, The Netherlands.
Non Verbal Communication What does the following sign mean to you?
Module 16 Emotion.
Consistent Dynamic-group Emotions for Virtual Agents. Abstract The use of computational models of emotion in virtual agents enhances the realism of these.
1 User experience User interfaces Jaana Holvikivi Metropolia.
COGNITIVE SCIENCE 17 Why Emotions Are Necessary Jaime A. Pineda, Ph.D.
Emotion-Driven Reinforcement Learning Bob Marinier & John Laird University of Michigan, Computer Science and Engineering CogSci’08.
Emotion Psychology, 4/e by Saul Kassin CHAPTER 12: Emotion 4/12/2017
A cognitive theory for affective user modelling in a virtual reality educational game George Katsionis, Maria Virvou Department of Informatics University.
Affective Computing: Machines with Emotional Intelligence Hyung-il Ahn MIT Media Laboratory.
Joost Broekens, Doug DeGroot, LIACS, University of Leiden, The Netherlands Emergent Representations and Reasoning in Adaptive Agents Joost Broekens, Doug.
Measuring emotions in an intergroup context and beyond Dr Roger Giner-Sorolla Department of Psychology Presentation given at the Research Methods Festival,
Emotion What Is an Emotion? Organized psychological and physiological reactions These reactions are: –Bodily Reaction –Affective (subjective experience)
GERG Santorini, June, 2004 “Ekman versus Scherer” or “What is important to recognize/express emotion?” (Selected GERG studies directed by Susanne.
Joost Broekens, Doug DeGroot, {broekens, LIACS, Leiden University, The Netherlands Scalable Computational Models of Emotion for Virtual.
Qualitative Feel and Emotion Nancy Alvarado. What is Emotion? Emotion has four aspects: Feelings (feels, qualia) Biological states – physiological arousal.
Affective Computing Lecture 5: Dr. Mark Brosnan 2 South:
Modeling Emotion Frameworks Useful In Computation and Speech Frank Enos.
Affective Computing and Human Robot Interaction a short introduction a short introduction Joost Broekens Telematica Institute, Enschede, LIACS, Leiden.
Emotion: More Than a Feeling
1 IUT de Montreuil Université Paris 8 Emotion in Interaction: Embodied Conversational Agents Catherine Pelachaud.
Emotions: Emotions: Hampson, E., van Anders, S. M., & Mullin, L. I. (2006). A female advantage in the recognition of emotional facial expressions: test.
The Discrete Emotions Theory Controversy in Psychology and Relevance to Consumer Behavior Louis Daily, Fiona Sussan, and Norris Krueger University of Phoenix.
Recognizing Emotions in Facial Expressions
Emotion, Stress, and Health chapter 13. Overview Nature of emotion Emotion and culture Nature of stress Stress and emotion How to cope chapter 13.
otivation ** Start of activity to meet physical or psychological need
From cognitive biases to panic: Modeling the mechanisms of anxiety disorders Eva Hudlicka Psychometrix Associates / U.Mass - Amherst Amherst, VA
COMPUTATIONAL MODELING OF INTEGRATED COGNITION AND EMOTION Bob MarinierUniversity of Michigan.
1 Modeling Emotions in Game Characters: Theoretical Foundations & Practical Guidelines Joost Broekens SEP 10, 2009 ACII 2009 Amsterdam Eva Hudlicka.
Affective computing and interface design measuring and modeling emotions for CHI Joost Broekens Delft University ERGOIA 2009 Workshop.
Computational Models of Emotion and Cognition Computational Models of Emotion and cognition Christopher L. Dancy, Frank E. Ritter, Keith Berry Jerry Lin,
IF A MAD SCIENTIST WERE TO REPLACE YOUR BEST FRIENDS BRAIN WITH A COMPUTER- HOW WOULD YOU KNOW SOMETHING WAS DIFFERENT? Emotion.
Emotion is Multifaceted Emotion refers to the mix of: 1. Physiological Arousal 2. Expressive Behaviors (how you react to the physiological arousal) 3.
Module 16.1 Perceiving Others. Copyright © Houghton Mifflin Company. All rights reserved Module 16.1 Preview Questions What is social perception?
5 Leadership Mind and Heart. Chapter Objectives Recognize how mental models guide your behavior and relationships. Engage in independent thinking by staying.
GUI: Specifying Complete User Interaction Soft computing Laboratory Yonsei University October 25, 2004.
Perception in management Dr Joan Harvey. Sensation: precedes perception and concerns the basic senses  vision  hearing  kinaesthesis  touch  smell.
Affective Computing: Agents With Emotion Victor C. Hung University of Central Florida – Orlando, FL EEL6938: Special Topics in Autonomous Agents March.
Computational Investigations of the Regulative Role of Pleasure in Adaptive Behavior Action-Selection Biased by Pleasure-Regulated Simulated Interaction.
4/12/2007dhartman, CS A Survey of Socially Interactive Robots Terrance Fong, Illah Nourbakhsh, Kerstin Dautenhahn Presentation by Dan Hartmann.
1 Student Motivation and Affect EDC 312 Chapter 6 Dr. Diane Kern.
BIM 7 Review: Numbered Heads Together Name 5 practical suggestions of ways teachers can access the social states of the brain: Name and define the PIES.
Integrating high- and low-level Expectations in Deliberative Agents Michele Piunti - Institute of.
Physical Emotion Induction and its use in entertainment Ralph Kok 1, Joost Broekens 2 1 Media Technology Master programme, Leiden University, the Netherlands.
Balancing Exploration and Exploitation Ratio in Reinforcement Learning Ozkan Ozcan (1stLT/ TuAF)
Bob Marinier Advisor: John Laird Functional Contributions of Emotion to Artificial Intelligence.
Design Principles for Creating Human-Shapable Agents W. Bradley Knox, Ian Fasel, and Peter Stone The University of Texas at Austin Department of Computer.
Learning to Navigate Through Crowded Environments Peter Henry 1, Christian Vollmer 2, Brian Ferris 1, Dieter Fox 1 Tuesday, May 4, University of.
Emotion An introduction and approaches to understanding T. Curwen.
Module 16 Emotion.
ATTENDING MODELING of End-of-Life Care James Hallenbeck, MD VA Palo Alto HCS.
SAD, ANGER, FEAR, DISGUST, HAPPY, CONTEMPT AND OTHERS ARE EVOLUTIONARILY DICTATED ADAPTIVE SURVIVAL MECHANISMS Darwin Tompkins Ekman Izard.
MIT Artificial Intelligence Laboratory — Research Directions The Next Generation of Robots? Rodney Brooks.
Emotional Intelligence
International Forum on English Language Teaching, University of Porto Ana Fernández Viciana University of Oviedo.
From Imprinting to Adaptation: Building a History of Affective Interaction Adaptive System Research Group School of Computer Science University of Hertfordshire,
Child & Young Person Development TDA 2.1 Session 2.
Attitudes and Attitude Change
Discuss the extent to which cognitive and biological factors interact in emotion.
Emotion and Sociable Humanoid Robots (Cynthia Breazeal) Yumeng Liao.
The problem. Psychologically plausible ways of
Think about the skill you picked for this semester
Chapter 9 Lesson 3 Section 4: Emotion.
Cooperation within Groups
Lecture 5: Dr. Mark Brosnan 2 South:
Expressing and Experiencing Emotion
Computational Aspects of Emotion in Adaptive Behavior
Module 16 Emotion.
Social and Emotional Development.
Presentation transcript:

Affective Facial Expressions Facilitate Robot Learning Joost Broekens Pascal Haazebroek LIACS, Leiden University, The Netherlands

Why? Interactive robot learning –Facilitate human-robot interaction –Study learner-teacher relations –Study learning and adaptation –Future: enable robots to interactively cooperate in an efficient manner with humans in ways that are natural to humans. –This talk: human affective facial expressions as training signal to robot

Outline Emotion influences thought and behavior EARL: –Studying relation emotion and adaptation in reinforcement learning This talk: Human affect as reinforcement to robot Reinforcement-based robot learning Experiments: –Affect as additional reinforcement –Affect as input to social reward function Results and conclusions –Learning is positively influence, especially in the learned social reward function case

Emotion, Thought and Behavior Emotion –Bodily expression (face, posture) –Action tendencies (Frijda) –Feelings –Cognitive appraisal (Arnolds, Lazarus, Scherer) Affect –Everything to do with emotion, as in affective computing, or –abstraction over emotion (e.g., Russell) composed of Arousal (alertness) Valence (pleasure) –We use the latter definition of affect in the experiment Short timescale Ignore arousal

Emotion, Thought and Behavior Emotion and affect influence thought and behavior: –The kind of thoughts we have Mood congruency –The way we process information Narrow vs. broad look (Goschke & Dreisbach) A lot vs. a little processing effort (Scherer, Forgas) –What we think about things Emotion/mood as information (Clore & Gasper) Emotion as belief anchor (Frijda & Mesquita) –How we learn and adapt Emotion/affect as social reinforcement Emotion/affect as intrinsic reinforcement Emotion as metaparameter to control learning process Empathy

EARL To study relations between emotion and adaptation in context of reinforcement learning. –Simulated robot (but see later comments) –Maze navigation tasks –Webcam and emotion recognition to interpret emotions –Reinforcement learning (RL) approach to robot learning –Robot has own model of emotion –Robot head to express emotion Potential influences experimented with –Evaluate models of emotion in RL setting –Evaluate models of emotional expression. –Test influence of emotion/affect on RL learning parameters –Experiment with communicated and robot emotion as reward

EARL Short movie

Human Affect as Reinforcement to Robot Interactive robot learning –Learning by Example E.g., imitation learning (see Breazeal & Scassellati) –Learning by Guidance (Thomaz & Breazeal) Future directed learning cues Anticipatory reward –Learning by Feedback Additional reinforcement signal (Breazeal & Velasquez; Isbell et al; Mitsunaga et al; Papudesi & Hubert) In our experiment: affective signal as additional reinforcement

Human Affect as Reinforcement to Robot Affective signal as additional reinforcement –Web cam –Emotional expression analysis –Positive emotion (happy) = reward –Negative emotion (sad) = punishment –So: emotional expression is used in learning as r human, a social reward coming from the human observer Note: –We interpret happy as positively valenced and sad as negatively valenced. –THIS IS A SIMPLIFIED SETUP THAT ENABLES US TO TEST OUR HYPOTHESIS!

Reinforcement-based robot learning Continuous Gridworld –World features placed on grid –Agent has Real coordinates and speed, unlimited locations –Local perception, agent-based perspective, = current state s Task –find food (as usual) Training: Multilayer perceptron networks (MLP) –Input is agents perceived state, s. –Each action (fwd, left, right) has two networks First to train action-value Q a (s) Second to train inverse action-value (value of NOT doing the action) –Value function has network trained to predict Q(s) Action-selection uses action values as predicted by MLPs In terms of representing the world, the perceived state and the actions, this setup is close to real-world robotics.

Reinforcement-based robot learning pathwall food 1/e 2 b

Experiments: Affect as additional reinforcement Test difference between standard agent and social agents –200 trials to learn path to food –Standard agent uses R(s) from environment to update Q a (s) and Q(s). –Social agent uses r human in addition to R(s)

Experiments: Affect as additional reinforcement Three social settings –Moderate social reinforcement (setting a) r human is small Long period of training with r human (trials 20-30) –Strong social reinforcement (setting b) r human is large Short period of training with r human (trials 20-25) –Learned social reinforcement (setting c) r human is used as above and to train R social (s) (an MLP). Period using r human is between 29 and 45. After that, R social (s) is used.

Results Moderate social reinforcement

Results Strong, short, social reinforcement

Results Learned social reinforcement

Conclusion A critical learning period can be used to influence robot learning using affective signals, in real-time, in a non-trivial learning environment. This has a benefit on learning Most specifically when the robot learns to predict the social feedback by training a reward function R social (s)

Further work Use affect/emotion as metaparameter to control –Learning rate –Exploration exploitation Differentiate between meanings of negative and positive emotions –Anger: negative feedback due to action of agent –Fear: negative anticipatory feedback –Surprise: strong positive feedback due to action of agent –Frustration: connect to exploration/exploitation rate? Affective Robot-Robot interaction? Use robot to human signals such as hesitation