1. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Making the head smile Smile? -> part of non-verbal communication Functions of non-verbal communication To enhance linguistic information To give new information

2. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Non-verbal communication Figure by Jari Kätsyri

3. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Voice prosody Figure by Jari Kätsyri

4. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Head Figure by Jari Kätsyri

5. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Eyes Figure by Jari Kätsyri

6. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Facial expressions Figure by Jari Kätsyri

7. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Gestures Figure by Jari Kätsyri

8. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Conversation regulation Figure by Jari Kätsyri

9. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Conversational signals Figure by Jari Kätsyri

10. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Maintenance Figure by Jari Kätsyri

11. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Emotions Figure by Jari Kätsyri

12. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Expressing emotions Figure by Jari Kätsyri

13. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Conversational signals Figure by Jari Kätsyri

14. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Voice Figure by Jari Kätsyri

15. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Head Figure by Jari Kätsyri

16. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Eyes Figure by Jari Kätsyri

17. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Face Figure by Jari Kätsyri

18. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Examples

19. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Dimitris Metaxas, Normal Badler Modeling of hand gestures Modeling emotions, personality…

20. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Facial expressions Emotional expressions in face Are expressions showing emotions? How they can be described? What kind of emotions are there?

21. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Emotion in face: To communicate or to express? Communicating vs. expressing emotions Intentional communication (Chovil) vs. Involuntary behaviour (Ekman) Technology viewpoint Emotion = Emotional expression = (Non- verbal) communication (In most situations emotions inappropriate)

22. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Emotions and facial expressions (Ekman) Emotion  Facial expression ? Emotional expressions can be inhibited No evidence for non-expressed emotions Facial expression  Emotion ? Emotional expressions can be faked There are facial muscles with no voluntary control Facial expression  Emotion (?)

23. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Description of expression Several descriptions of expressions exist: Facial Action Coding System – FACS By Eckman and Friesen ”Language of facial expressions” MPEG-4 Synthetic-Natural hybrid coding (SNHC) “Technical” description for low-bandwidth transmission

24. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 FACS Noticeable changes, not exactly facial muscles Tool for identifying facial actions Not for modeling facial actions Does not describe dynamics Action units (AUs) AU1: Inner brow raiser … AU23: Lip tightener … AU66: Crosseye

25. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 FACS (Facial Action Coding System) Tool for identifying facial actions Action Units Not for modelling facial actions

26. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 AU examples: AU7 (lid tightener)

27. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 AU examples: AU9 (nose wrinkler)

28. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 AU examples: AU11 (nasolabial furrow deepener)

29. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Pure and blended emotions 7 pure emotions Pleasure (pl), surprise (su), fear (f), hate (h), sorrow (so), disgust (d), (interest) Blended emotions Combinations of pure emotions Easy: pl+su, h+d, so+f Difficult: su+f, d+pl, h+so, f+pl (Some) additivity

30. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Possible causes of blended expressions in real world Mixed emotions (sadness+disgust) Consequent emotions (surprise+happiness) Masking (anger+happiness) …

31. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Attempts to add emotions

32. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 neutral sad surprise

33. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 6 basic expressions courtesy Jari Kätsyri

34. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 courtesy Jari Kätsyri Blended expressions

35. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Textured × non-textured Identification of expressions from pictures and videos of textured and non-textured version of AP

36. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Differences among databases Number of correct responses for pictures and videos of different databases

37. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Neutral expression Average evaluations for AP textured and non- textured neutral expressions

38. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Better implementation Collecting data

39. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Collecting data for expressions happy anger

40. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 HUT facial expression database First Finnish database of moving facial expressions

41. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Study: Interaction of blur x type x emotion New database, 6 actors 3 levels of blur Pictures, short video sequences 6 basic expressions 3 subject groups (17 each) each group – 1 level of blur each actor 6 expressions 3 actors pictures, 3 actors videos

42. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Interaction of blur x type x emotion

43. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 The end Thank you! and … a little exercise again