Affective computing and interface design measuring and modeling emotions for CHI Joost Broekens Delft University ERGOIA 2009 Workshop.

Slides:



Advertisements
Similar presentations
1 Arousal and Emotion Whats their use?? Assist in decision making Readiness.
Advertisements

Affective Facial Expressions Facilitate Robot Learning Joost Broekens Pascal Haazebroek LIACS, Leiden University, The Netherlands.
Computational Aspects of Emotion in Adaptive Behavior Joost Broekens, Walter Kosters, Fons Verbeek LIACS, Leiden University, The Netherlands.
Cognitive Systems, ICANN panel, Q1 What is machine intelligence, as beyond pattern matching, classification and prediction. What is machine intelligence,
Cross Cultural Research
Addressing Patient Motivation In Virtual Reality Based Neurocognitive Rehabilitation A.S.Panic - M.Sc. Media & Knowledge Engineering Specialization Man.
Emotion Psychology, 4/e by Saul Kassin CHAPTER 12: Emotion 4/12/2017
Emotion Chapter 11 Emotion 4/12/2017
Agenda for February 24 Announcements: Mosaic conference Dimensions events Smith & Ellsworth’s Cognitive Appraisal Theory Presentation by Discussants Review.
A cognitive theory for affective user modelling in a virtual reality educational game George Katsionis, Maria Virvou Department of Informatics University.
Relating Error Diagnosis and Performance Characteristics for Affect Perception and Empathy in an Educational Software Application Maria Virvou, George.
PSYC 1000 Lecture 44. Emotion –Response of whole organism to pleasant and aversive events of different types Happiness, Sadness, Fear, Anger, … –Three.
Emotion induction techniques Geneva Emotion Research Group.
To what extent do biological and cognitive factors interact?
Joost Broekens, Doug DeGroot, {broekens, LIACS, Leiden University, The Netherlands Scalable Computational Models of Emotion for Virtual.
1 Affective Learning with an EEG Approach Xiaowei Li School of Information Science and Engineering, Lanzhou University, Lanzhou, China
Affective Computing Lecture 5: Dr. Mark Brosnan 2 South:
Based on a fine paper byPhilippe Zimmermann
ISTD 2003, Thoughts and Emotions Interactive Systems Technical Design Seminar work: Thoughts & Emotions Saija Gronroos Mika Rautanen Juha Sunnari.
Modeling Emotion Frameworks Useful In Computation and Speech Frank Enos.
Affective Computing and Human Robot Interaction a short introduction a short introduction Joost Broekens Telematica Institute, Enschede, LIACS, Leiden.
Usability 2004 J T Burns1 Usability & Usability Engineering.
Evolution Universals v. Diversity. Battle of Universals and Cultures Human universals: Search for unifying parameters of functioning –Emphasizes biology.
1 A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions Zhihong Zeng, Maja Pantic, Glenn I. Roisman, Thomas S. Huang Reported.
Principles of High Quality Assessment
Meaningful Learning in an Information Age
1 IUT de Montreuil Université Paris 8 Emotion in Interaction: Embodied Conversational Agents Catherine Pelachaud.
Emotion, Stress, and Health chapter 13. Overview Nature of emotion Emotion and culture Nature of stress Stress and emotion How to cope chapter 13.
Chapter 4 Learning: Theories and Program Design
Usability 2009 J T Burns1 Usability & Usability Engineering.
1 Modeling Emotions in Game Characters: Theoretical Foundations & Practical Guidelines Joost Broekens SEP 10, 2009 ACII 2009 Amsterdam Eva Hudlicka.
Journalism 614: Attitudinal Perspectives on Opinion Expression.
Computational Models of Emotion and Cognition Computational Models of Emotion and cognition Christopher L. Dancy, Frank E. Ritter, Keith Berry Jerry Lin,
Module 16 Emotions Kimberly, Diana, Kristen, JP, Chris, Michael, Chris.
Understand the sequence of oral presentation assignment components Learn how to develop explanations for assigned material –Listen to lecture on Rowan.
Emotional Machines Presented by Chittha Ranjani Kalluri.
Emotion Module 12. What are emotions? full body responses, involving: 1. physiological arousal (increased heart rate) 2. expressive behaviors (smiling,
Chapter 8: Motivation and Emotion
Emotion. It is a big concept, not easy to define. Drever (1964) Emotion involves ‘bodily changes of a widespread character- in breathing, pulse, gland.
Emotion.
GUI: Specifying Complete User Interaction Soft computing Laboratory Yonsei University October 25, 2004.
THE IMPACT OF COMPUTER SELF-EFFICACY AND TECHNOLOGY DEPENDENCE ON COMPUTER-RELATED TECHNOSTRESS: A SOCIAL COGNITIVE THEORY PERSPECTIVE Qin Shu, Qiang Tu.
11.10 Human Computer Interface www. ICT-Teacher.com.
Enhancing Teaching and Learning with Podcasts Mico e-Learning Workshop.
 Learning is acquiring new or modifying existing knowledge, behaviors, skills, values or preferences and may involve synthesizing different types of.
Bob Marinier Advisor: John Laird Functional Contributions of Emotion to Artificial Intelligence.
©1999 Prentice Hall Emotion Chapter 11. ©1999 Prentice Hall Emotion Defining Emotion. Elements of Emotion 1: The Body. Elements of Emotion 2: The Mind.
Module Nine: Emotional Communication (Conversation) 8- 1.
Emotion. Emotion  It is at the heart of who we are as people  It often is a reflection of our mental state  We are going to look at different theories.
Chapter 13 The Subjective and Physiological Nature of Emotions.
Motivation and Emotion. Motivation Motivation - process by which activities are directed so that physical or psychological needs/wants are met. Extrinsic.
Mindful Shift Chapter 12. “Of all species on earth, we human have the capacity of mind change: we change our minds and that of others”
Chapter 1: What is interaction design?. Bad designs From:
Toward a Unified Scripting Language 1 Toward a Unified Scripting Language : Lessons Learned from Developing CML and AML Soft computing Laboratory Yonsei.
EEL 5708 Affective computing Lotzi Bölöni. EEL 5708 Affective computing Emotions and computers. Initiator of the field: Rosalind Picard at MIT –
Module 16 Emotion.
Emotion. Emotion Defining Emotion Defining Emotion Elements of Emotion 1: The Body Elements of Emotion 1: The Body Elements of Emotion 2: The Mind Elements.
Motivation and Emotion. Motivation Motivation - process by which activities are directed so that physical or psychological needs/wants are met. Extrinsic.
Emotional Intelligence
Chapter 7 Affective Computing. Structure IntroductionEmotions Emotions & Computers Applications.
Copyright © 2013, 2010, 2007 Pearson Education, Inc. All Rights Reserved.
Emotions. Emotion A state of arousal involving facial and body changes, brain activation, cognitive appraisals, subjective feelings, and tendencies toward.
EMOTION BY: JORDAN, MATT, DOUG, AND JORDAN. WHAT IS EMOTION? Emotion- a natural instinctive state of mind deriving from one's circumstances, mood, or.
WP6 Emotion in Interaction Embodied Conversational Agents WP6 core task: describe an interactive ECA system with capabilities beyond those of present day.
The Three Domains of Physical Education. What does Physical Education mean to you?
Interpreting Ambiguous Emotional Expressions Speech Analysis and Interpretation Laboratory ACII 2009.
Emotion and Sociable Humanoid Robots (Cynthia Breazeal) Yumeng Liao.
The problem. Psychologically plausible ways of
Lecture 5: Dr. Mark Brosnan 2 South:
Computational Aspects of Emotion in Adaptive Behavior
Presentation transcript:

Affective computing and interface design measuring and modeling emotions for CHI Joost Broekens Delft University ERGOIA 2009 Workshop

Outline Emotion and affect in human behavior Affect measurement and recognition Affect representation and modeling Applications: overview + two detailed examples

Emotion and affect in human behavior Basic emotions: fear, anger, happiness, sadness, surprise, disgust Short episode of synchronized system activity triggered by event: – subjective feelings (the emotion we normally refer to), – tendency to do something (action preparation), – facial expressions, – evaluation of the situation (cognitive evaluation, thinking), – physiological arousal (heartbeat, alertness). Affect = related to emotion, mood and attitudes: – emotion: object directed, short term, high intensity, action oriented, differentiated. – mood: unattributed, undifferentiated, longer term, low intensity. – attitude: affect permanently associated with an object/person – affect: abstraction of emotion/mood in terms of, positiveness/negativeness and activation/deactivation (e.g., Russell, Rolls).

Emotion and affect in human behavior Situational evaluation and communication. Heuristic relating events to actions through an evaluation of personal relevance (e.g., goals, needs) : – Evaluation of personal relevance of event (Scherer) – Speeds-ups decision-making (Damasio) – fast reactions and action preparation (Frijda) – influence information processing (Isen, Forgas) Learning & adaptation, attention, mental search/planning, creativity, etc.. Communication medium: – communicate internal state (Darwin, Ekman) – alert others – show empathy (understanding of situation of others).

Emotion: dimensions

Emotion: categories Sadness: – Low arousal – Face: sad – Avoid – Bad feeling Anger: – High arousal – Face: angry – Approach – Bad feeling Joy: – High arousal – Face: happy – Play – Good feeling Category is a typical “emotion syndrome” – A complex of physiology, expression, behavior, and feeling

Emotion: components Stimulus checks – (Scherer: cognitive appraisal theory) NoveltyPleasantnessGoal/Need conduciveness Coping potential Sensory- Motor level Sudden, intense stimulation Innate preferences/ aversions Basic needsAvailable energy Schematic level Familiarity: schema matching. Learned preferences or aversions Acquired needs motives Body schema Conceptual level Expectations: cause/effect, probability Recalled, anticipated, or derived positive- negative estimates Conscious goals, plans Problem-solving ability.

Emotion: summary

Emotion and affect in human behavior Many relations between affect and cognition: Mood influences information processing style – Top-down (positive) versus bottom-up (negative) – Heuristic/generic/assuming/creative processing (positive) versus detail/feature/critical/procedural processing (negative) Mood influences learning – Flow, boredom, frustration, etc. Emotion influences information processing – Strong (arousing) emotions hamper processing in general.

Emotion and affect in human behavior Attitudes influence information processing – Strong attitudes stop search E.g., a strong negative association with an option discards it – Attitudes influence exploration direction E.g., a low intensity negative association biases search against that direction. Affective influence depends on processing style – Direct access (weak influence) – Heuristic (strong influence) – Procedural (weak influence) – Elaborate (strong influence)

Can computers/robots use emotion in a constructive sense? To communicate with humans? –Animal emotions evolved for communication purposes To be more adaptive? –Animal emotions evolved for adaptive purposes as well To better understand / adapt to humans? As modeling tool to simulate and understand human emotions better? –The computer is a medium to simulate a theoretical model. This field of research is called Affective Computing (see also the book by Rosalind Picard) Please note: this is not emotional design

Affective Computing Computing that relates to, arises from, or deliberately influences emotions (Picard, 1997). Different types of computational approaches: – recognize or measure human emotions (recognition). – interpret human emotion (perception, processing). – represent human emotion – elicit emotions (cognitive modeling, motivations, feedback). – represent system emotion. – emotional influence on behavior and functioning (adaptation, attention, actions). – show system emotions (expression). – Influence human emotion (induction). Form not important: a robot, a virtual character, a tutor agent, a fridge, etc…

Affect measurement and recognition

Affect measurement and recognition: why? Living Lab experiments – Evaluate products, test hypotheses about emotion theory, etc. Social software – Human communication, expression, etc. Software that uses affect feedback for functioning – Recommendation, (serious) games, tutor agents, VR training, etc.

Affect measurement and recognition: how? Implicit (automated affect recognition) – Physiological: Galvanic Skin Response, Heart rate, muscle tone, EEG – Behavior-based: Facial expression analysis, body posture, gestures, sound, speech, mouse movement, keyboard presses. Issues – Deception/ Display rules – Ambiguity (context) and precision/range – Noise – Positioning – Invasiveness – One modality problematic (multi-modal needed) – Time-scales – Type of affect recognized (mood/emotion/mixed/intensity?)

Examples of implicit feedback

Affect measurement and recognition: how (2)? Explicit (affective feedback) – Ask affective feedback Free text, questionnaires, emotion words, experience sampling, experience clips – Affect dimension-based Affect questionnaires, SAM, AffectButton, prEmo, EmoCards, etc. – Facial-expression-based Emoticons, basic emotion icons, etc. – Text-based (actual in between explicit and implicit): websites, blogs, documents, tags – Haptics SEI, EmoPen, Emoto Issues – Verbal report – Subjective interpretation bias / cultural bias – Validity and reliability. – Deception / social conformation – Ambiguity (context) and precision/range – Useability/learnability – Type of affect recognized (mood/emotion/mixed/intensity?)

Examples of explicit feedback Self-Assessment Manikin (SAM) (Bradley&Lang 1994) Purely dimension-based (Please Arousal Dominance)

Examples of explicit feedback (Sanchez et al 2006) Dimension-based + labels (Pleasure, Arousal, Dominance)

Examples of explicit feedback EmoCards (Desmet, 2001) Dimension-based + labels (Pleasure, Arousal)

Examples of explicit feedback Experience drawing (Tahti & Arhippainen, 2004) Bounded form of experience expression by user.

Examples of explicit feedback Haptic feedback (Smith & MacLean, 2007) Sensual Evaluation Instrument (Hook et al, 2005)

Examples of explicit feedback Affective gestures (Fagerberg, Stahl, Hook, 2004) Accelerometer and a pressure sensor attached to stylus pen.

Affect representation and modeling

How to represent (human) affect in a system? Remember: different views on emotion – Dimensional (valence, arousal, dominance) – Categorical (happy, angry, sad, etc.) – Componential(novelty, attribution, agency, etc.) Use these views as representational basis.

Emotion: dimensions Extract Pleasure, Arousal, Dominance from input signal, e.g., In text (e.g. websites, blogs): Map words to PAD using empirical date, integrate triples. In video/images/speech/physiological (e.g., movies, foto’s): Correlate features to PAD, or classify objects in +/- Explicit (interface component): Directly ask dimensions (SAM), use mapping from faces to PAD. Key benefit: easy to compute with, mixed emotions make sense Key problem: ambiguity and specificity

Emotion: categories Sadness: – Low arousal – Face: sad – Avoid – Bad feeling Anger: – High arousal – Face: angry – Approach – Bad feeling Joy: – High arousal – Face: happy – Play – Good feeling Extract emotion categories from input signal, e.g., In text (e.g. websites, blogs): Map words to Happy, Sad, Angry, etc.. using empirical date, integrate emotion vector, select most important one. In video/images/speech/physiological (e.g., movies, foto’s): Classify objects in emotion categories Explicit (interface component): Directly ask emotions Key benefit: easy to understand for users and developers Key problem: computation with mixed emotions and intensities

Emotion: components Ask user for explanation Extract goals, needs, desires from human Interpret situation and context Derive emotion from the above using appraisal theory. See e.g., the GATE project (Wherle, Kaiser, Scherer, etc.) Key benefit: detailed emotion Key problem: not many approaches exist, not clear how all this should be done NoveltyPleasantnessGoal/Need conduciveness Coping potential Sensory-Motor level Sudden, intense stimulation Innate preferences/ aversions Basic needsAvailable energy Schematic level Familiarity: schema matching. Learned preferences or aversions Acquired needs motives Body schema Conceptual level Expectations: cause/effect, probability Recalled, anticipated, or derived positive- negative estimates Conscious goals, plans Problem-solving ability.

Affect representation and modeling Keep in mind: We talked about measured/derived human affect But affect representation is equally important for a system/robot/agent that simulates/generates affect/emotion/mood – Emotional robots – Emotional NPC’s and Tutor agents Emotion generation will not be discussed in this presentation.

Applications

What to do with the emotion? Feedback and communication – feedback to learning system/robot (Broekens, 2007: EXPLAINED IN DETAIL LATER) – robot communication (Breazeal) Persuasive design – in VR training, tutor agents (Gratch & Marsella, Nijholt) – Treatment of emotion-related disorders such as ASD (de Silva et al, 2007) – emotions in simulated-agent plans (e.g., human-like reasoning) (Gratch & Marsella), – robot acceptance (Heerink) Affect-based adaptation – Affect-adaptive gaming and entertainment (Hudlicka, Yannakakis, Gilleade & Dix) – Affect-based music adaptation (Livingstone & Brown) – Emotional tagging and rating in recommenders (LeSaffre et al 2006) – Interactive TV (Hsu et al, 2007) Analysis and design – Web-site analysis (Grefenstette et al, 2004) – Inform design process (Desmet, Hook) – Living labs (Mulder) Etc…

Kismet (Breazeal) Social: Kismet, A framework, using a humanoid head expressing emotions, to study: –effect of emotions on human-machine interaction. –learning of social robot behaviors during human-robot play. –joint attention.

Companion Robots Aibo (Sony, Japan) Entertainment robot I-Cat (Philips, NL) Robot assistant for elderly people Paro (Wada et al, Japan) Robot companion for elderly Huggable (MIT, USA) Robot companion for elderly

SIMS 2 (Electronic Arts) Entertainment: emotions are used to provide entertainment value.

Mission Rehearsal Exercise (Gratch & Marsella) Cognitive: study the influence of artificial emotions on –planning mechanism of virtual characters, –training effect on trainees (emotion might enhance effect)

Virtual Training and Virtual Therapy Therapist skill training using virtual characters (Kenny et al, left) Social phobia training (at TU Delft, right)

HRI Application: Interactive Robot Learning

Interactive robot learning in short… A special case of Human Robot Interaction – Goal HRI: more efficient, flexible, personal, pleasant human-robot interaction Interactive Learning – Show examples of behavior to robot. – Direct learning process by guidance, and – by feedback. Why study this? – Robot perspective Facilitate human-robot interaction Study learning and adaptation – Human perspective Study learner-teacher relations

Reinforcement-based robot learning pathwall food Food (+) Agent Wall (-) Path Start Reward r maze = (+|-) feedback from the environment about action of robot. Learn by repetition which sequence of actions gives best positive feedback.

Experimental setup A Simulated learning robot in a Simple maze learning task (find shortest path to food) Webcam and emotion recognition to interpret human emotions Real-time affective feedback

Human affective feedback Normal learning feedback: – r maze from maze based on taken actions (+ = repeat, -=don’t repeat). Affective feedback: – In addition to feedback r maze from maze, – the expression is used in learning as a social reward r human Positive emotion = reward = + r human Negative emotion = punishment = - r human Real-time affective feedback

Experiment Test difference between standard agent and social agents Control condition: – Standard agent uses just r maze. Two social agents that use r human in addition to r maze : – Direct social reinforcement: r=r maze +r human – Direct and Learned social reinforcement: r=r maze +r human Robot learns to predict r human and, uses learnt feedback as surrogate r human when human stops giving feedback.

Results Direct social reinforcement Emotional feedback helps learning but effect goes away when human stops giving feedback. Why? Steps needed to find the food Number of times the food was found (successful trials)

Results Direct and Learned social reinforcement Again, emotional feedback helps learning and the effect stays.  it learned the feedback and keeps using this even when the human is away. Steps needed to find the food Number of times the food was found (successful trials)

HRI experiment: conclusion Affective signals can be used to train, in real-time, robot behavior. This has a measureable benefit on learning. Most specifically when the robot learns to predict the human feedback r human and uses that when the human is gone. But: did we express an emotion?

Emotion Measurement AffectButton: user friendly affect feedback

AffectButton: Why? Pleasure-Arousal-Dominance-Based Feedback – Data is “computation friendly” and continuous Static element in interface – No unfolding, easy to place in an interface Easy to use Easy to learn – Emotion selection time < 5 sec Valid and reliable feedback – Users agree on meaning of button, and are consistent.

AffectButton: experiment Users match a given emotion word with the AffectButton Emotion word has validated PAD values (Mehrabian, 1980) Use these values to correlate with user feedback Example: – Happy (p=.8, a=.4, d=.5) – Face in AffectButton should be selected matching these values

Validity and Reliability Validity: – Concurrent validity between feedback by users, and – Existing P, A, D scores for words. – Correlate – P =.9, A=.8, D=.81 Reliability: cronbach! – Inter-rater consistency: users are assumed to be raters – alpha is used as measure of agreement between raters for each emotion word. – Alpha was 0.97, 0.94, and 0.96 for Pleasure, Arousal and Dominance respectively

Problems/Questions! What did we measure? – Own feeling about word? Attitude about word? – What about mood induction influences? How to further evaluate reliability and validity? – We need broader cultural coverage with respect to evaluation. – We need more subjects. – Does the AffectButton have face validity? Can we express all important emotions with it? – Problem: complex emotions are difficult (guilt, jealousy, happy-for) Suggestions welcome: to download and play with it:

Useful introductory sources To feel or not to feel: The role of affect in human-computer interaction (Hudlicka, 2003). – And the accompanying Special Issue in the same journal. A survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions (Zeng, Pantic, Roisman, Huang, 2009) Experimental evaluation of five methods for collecting emotions in field settings with mobile applications (Isomursu, Tähti, Väinämö, Kuuti, 2007)