1. Modeling Expressivity in ECAs
Catherine Pelachaud,
Maurizio Mancini
LINC - University of Paris 8

2. Behavior Behavior is related to the (Wallbott, 1998):
quality of the mental state (e.g. emotion) it refers to
quantity (somehow linked to the intensity factor of the mental state)
Behaviors encode:
content information (the ‘What is communicating’)
expressive information (the ‘How it is communicating’)
Behavior expressivity refers to the manner of execution of the behavior

3. Behavior Representation
Behavior= signal shape, movement, expressivity
Gesticon (Gesture Lexicon): Dictionary of behavior description (B. Krenn, H. Pirker OFAI)
Modalities:
face
hand and arm gesture
body movement and posture
gaze

4. Behavior Representation
Face:
facial expression
duration: onset, apex, offset
Gesture:
phases: preparation, pre-hold-stroke, stroke, post- hold-stroke, retraction
gesture shape and movement for each phase
Head:
head direction
head movement
Gaze
eye direction

5. Expressivity Dimensions
Spatial: amplitude of movement
Temporal: duration of movement
Power: dynamic property of movement
Fluidity: smoothness and continuity of movement
Repetitiveness: tendency to rhythmic repeats
Overall Activation: quantity of movement across modalities
Implemented for gesture and facial expression

6. Overall Activitation
Threshold filter on atomic behaviors during APML tag matching
Determines the number of nonverbal signals to be executed.

7. Spatial Parameter the reach
Amplitude of movement controlled through asymmetric scaling of
the reach
Space that is used to find IK goal positions
Expand or condense the entire space in front of agent

8. Temporal parameter
Determine the speed of the arm movement of a gesture's
meaning-carrying stroke phase
Modify speed of stroke
Stroke shift / velocity control of a beat gesture
Y position of wrist w.r.t. shoulder [cm]
Frame #

9. X position of wrist w.r.t. shoulder [cm]
Fluidity
Continuity control of TCB interpolation splines and gesture-to-
gesture
Continuity of arms’ trajectory paths
Control the velocity profiles of an action
coarticulation
X position of wrist w.r.t. shoulder [cm]
Frame #

10. Power Tension and Bias control of TCB splines; Overshoot reduction
Acceleration and deceleration of limbs
Hand shape control for gestures that do not need hand configuration to convey their meaning (beats).

11. Repetitivity
Technique of stroke expansion: Consecutive emphases are realized gesturally by repeating the stroke of the first gesture.

12. Expressivity Expressivity values may act:
over the whole animation: EmoTV, analysis- synthesis
every instant of the movement: Greta Music
on every gesture: GEMEP corpus
on a particular phase of the gesture: Attract attention study
Exploratory studies based on various data types:
acted data
real data
2D cartoon
literature

13. Research Issue Behavior representation Implementation refinement
what to encode
at which levels of representation
dynamism
Implementation refinement

14. Expressivity over the WHOLE animation
One set of values are set
extracted manually from annotation of real data video corpus
extracted automatically from video corpus of acted data using image analysis technique

15. Expressivity over the WHOLE animation
From annotations to animation Jean-Claude Martin, Laurence Devillers, LIMSI-CNRS; Maurizio Mancini, Paris8
Consider what is visible:
annotate signals, how there are displayed, how they are perceived
Model what is visible:
represent signals, animate them with expressivity
Two-steps approach
Elaborate rules by analysis (video corpus)
Animate by “copy synthesis”
No model of the processes underlying the display of the signals
Annotation  extraction  animation

16. Expressivity over the WHOLE animation
EmoTV: 51 clips French TV Interviews
Annotation
Emotion labels: single and blend of emotions
Multimodal behavior
Expressivity dimensions
Annotation
Steps
Extraction
Generation
Animation
EmoTV
clip
GRETA
animation

17. Expressivity over the WHOLE animation
Expressivity of gestures in mixed emotion
Expressivity parameter
Anger
Despair
AngerDespair from annotations
Temporal Extent 1 -1
Fluidity
0.58
Power
-0.5
0.11
Repetition
values obtained from literature
values obtained from annotation

18. Video

19. Expressivity over the WHOLE animation
Real World
Sensing
Virtual
Sensory
Storage
Perception
Generation
Attention
Planning
Interpretation
Personality
Scene
Ontology
Goals
Real and Virtual World Sensing A. Raouzaiou, G. Caridakis, K. Karpouzis ICCS; C. Peters, E. Bevacqua, M. Mancini, Paris 8

20. Expressivity over the WHOLE animation
Application Scenario

21. Expressivity over the WHOLE animation
Interpretation:
gesture specified by symbolic name
facial expression:
emotion label: if the facial expression corresponds to one of the prototypical facial expression of emotions
otherwise, FAPs values
Planning:
modulate expressivity parameters
module emotional expressions

22. Expressivity over the WHOLE animation
Generation
Input to ECA system:
a symbolic description of a gesture
emotion label or set of FAPs value
expressivity parameters value
Output: facial and gesture animation

23. Video

24. Expressivity on Every Frames
Greta Music Roberto Bresin, KTH -
Maurizio Mancini, Paris8
One set of expressivity paramters values are extracted automatically from acoustic data in real-time and fet to the ECA system.

25. Expressivity on Every Frames
Design a tool for real-time visual feedback to expressive performance

26. Expressivity on Every Frames
From music expression to facial expression
From acoustic cues to emotion : extraction of acoustic cues:
Tempo, Sound Level, Articulation (staccato/legato),
Attack Velocity, Spectrum, Vibrato rate, Vibrato
Extent, Pitch
From acoustic cues to facial expression: mapping of acoustic cues:
music emotion  facial expression
music volume  spatial and power
music tempo  temporal and overall activation
music articulation  fluidity

27. Expressivity on Every Frames
Music version of Greta
This version of Greta allows only the following actions:
head moving
eyes blinking
emotional expression
skin colouring
Input: Expressivity Parameters Output:
FAP values (animated head)

28. Expressivity on Every Frames
Video

29. Attraction of attention
Expressivity on Gesture Phases
Attraction of attention
Corpus: videos from traditional animation that illustrate different types of conversational interaction
Analysis: the modulations of gesture expressivity over time play a role in managing communication, thus serving as a pragmatic tool
France Telecom

30. Attraction of attention
Expressivity on Gesture Phases
Attraction of attention
Irregularities
the principle of anticipation:
it enhances the visibility of a gesture
it enhances our propensity to gaze at this gesture.
Discontinuities
create a contrast between successive gestures
function to isolate a particular gesture from a sequence of gestures
France Telecom

31. Expressivity on Gesture Phases
Irregularity

32. Irregularity – slow motion
Expressivity on Gesture Phases
Irregularity – slow motion

33. Discontinuity – slow motion
Expressivity on Gesture Phases
Discontinuity – slow motion

34. Application: ECA as web presenter Discontinuity – spatial parameter
Expressivity on Gesture Phases
Application: ECA as web presenter Discontinuity – spatial parameter

35. Expressivity on Each Gesture
Annotation of multimodal behavior
signals on 3 modalities
arm gesture
head movement
body movement
phases of each signal
each phase: physical shape + timing
expressivity of each signal

36. Annotation of multimodal behavior
Expressivity on Each Gesture
Annotation of multimodal behavior

37. Expressivity on Each Gesture
Animation format
we have defined a file format for the specification of behavior for our animation engine (Greta)
we translate from the XML annotation file (ANVIL) to the engine animation file
ANVIL
annotation
animation
file

38. Expressivity on Each Gesture
Animation format

41. Demo
demo!