Expressive Emotional ECA ✔ Catherine Pelachaud ✔ Christopher Peters ✔ Maurizio Mancini

Expressive Emotional ECA ✔ Emotion behavior modeling: ✔ Signals ✔ Temporal course of signals ✔ Expressive qualities ✔ Often in ECA systems: ✔ only modeling of signals is considered ✔ 6 ‘universal’ expressions

Large variety of facial expressions (Kollias et al) ✔ Archetypal expressions modeling using FAPs MPEG-4 ✔ Estimation of the range of variations of FAPs in archetypal expressions ✔ Intermediate expressions: rule-based computational model based on ✔ Whissell’s representation: ✔ Activation, evaluation, ✔ Plutchik’s angular measure

Laban Movement Analysis ✔ 5 major components: ✔ Body: part of the body in used ✔ Space: description of the directions and paths of motion ✔ Shape: changing forms of the body ✔ Effort: dynamics of the body moves ✔ Relationship: modes of interaction with oneself, others and the environment (facing, contact…) ✔ Textual and symbolic language of movement description

EMOTE (Badler et al) ✔ EMOTE model for Effort and Shape ✔ Based on Movement Observation Science ✔ Laban Movement Analysis ✔ Computational model of Effort and Shape components

Effort ✔ Four motion factors: space, weight, time, flow ✔ Space ✔ Weight ✔ Time ✔ Flow ✔ Each factor ranges from: ✔ Indulging in the quality ✔ Fighting against the quality

Effort ✔ Space: indirect  direct ✔ Waving away bugs  pointing to a particular spot ✔ Weight: light  strong ✔ Feather movement  punching ✔ Time: sustained  sudden ✔ Stretching  grabbing a falling object ✔ Flow: free  bound ✔ Waving widely  moving in slow motion

Shape ✔ Three distinct qualities of change in the form of movement: ✔ Shape flow: mover’s attitude toward the changing relationship among body parts ✔ Directional movement: mover’s intent to bridge the action to a point in the environment ✔ Shaping: mover’s carving or molding attitude with the environment

Shape ✔ Three dimensions: ✔ Horizontal: spreading  enclosing ✔ Opening arms to embrace  clasping someone in a hug ✔ Vertical: rising  sinking ✔ Reaching for something in a high shelf  stamping the floor with indignation ✔ Sagittal: advancing  retreating ✔ Reaching out to shake hand  Avoiding a punch

Expressive Communicative Gestures (Hartmann et al) ✔ Expressivity parameters: ✔ Spatial: amplitude of movement ✔ Temporal: duration of movement ✔ Power: dynamic property of movement ✔ Fluidity: smoothness and continuity of movement ✔ Repetitiveness: tendency to rhythmic repeats ✔ Overall Activation: quantity of movement across modalities ✔ Expressivity modeling: modal and intermodal

Emotional Modeling Carmen's Bright IDEAS ( Stacy Marsella, USC) Interactive drama Agent exhibits gestures based on their emotional states and personality traits

FantasyA (Ana Paiva, Inesc) Role-playing game The emotion detected from user drives the behavior of one of the agents in the video game

SenToy (Ana Paiva, Inesc) SenToy is a tangible interface device, used to influence emotional behaviour in FantasyA SAFIRA: EU project

Behavior Modeling ✔ Facial expression: ✔ Use facial analysis techniques/motion capture to look at: ✔ Value of facial animation parameters (muscular contraction, FAP…) ✔ Pattern of movement for a given parameter ✔ Relation of facial animation parameters within a same expression ✔ Coarticulation of movements between expressions

Mocap

Mocap

FAPs – MPEG-4

Example

Gesture ✔ Gesture specification often based on sign language gesture specification: ✔ Hand description ✔ Palm orientation ✔ Wrist position ✔ Movement: wrist, elbow, finger… ✔ Parameters characterizing quality of movement: Laban, sub-set…? ✔ Parameters act on overall animation (Emote) or on gesture phases (preparation, stroke, …)?

Behavior ✔ Gaze: ✔ Direction, gaze length, mutual gaze, ratio of gaze avoidance/gaze look at ✔ Body: ✔ Body posture, overall movement quality ✔ Voice: ✔ Paralinguistic parameters ✔ Text

Signals analysis/generation ✔ Use analysis method to: ✔ Get information on parameters ✔ Multimodal integration ✔ Signals synchrony across modalities ✔ Build computational model for generation ✔ Get the expressive quality of behavior ✔ Signals generation should not be done on one frame only but on a temporal interval

Conclusion ✔ Emotion is shown through: ✔ Signals ✔ Their temporal course ✔ Which parameters to look at? Which are the perceptually relevant ones? ✔ How to describe: ✔ The signals? ✔ Their evolution through time? ✔ Correlation between modalities?

Conclusion ✔ Relation between intensity of emotion and behaviors? ✔ Selection of behaviors ✔ Quality of behaviors ✔ Duration of behaviors ✔ Behaviors across modalities