1. Recognition, Analysis and Synthesis of Gesture Expressivity George Caridakis IVML-ICCS

2. humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis Overview Corpus Image processing module Gesture Recognition Expressivity Analysis Expressivity Synthesis Applications

3. humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis Overview

4. humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis Corpus mint-IVML 7 subjects 7 gesture classes 20 gesture variations (3 quadrants) 20’ minutes – 30000 frames

5. humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis Corpus EmoTV

6. humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis Corpus GEMEP (on going…)

7. humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis Head detection Detect candidate facial areas Validate using skin probability Conclude on number of persons

8. humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis Hand Detection Skin probability Thresholding & Morphology Operations Distance Transform Frame difference

9. humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis Tracking Scoring system based on: Skin region size Distance wrt the previous position Optical flow alignment Spatial constraints Thresholding scores Periodical re-initialization

10. humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis Head & Hand Tracking

11. humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis HMM parameters for gestures States are head and hands coordinates XL-XR XH-XR XH-XL YL-YR YH-YR YH-YL 6 output states Bakis left-to-right models Continuous output distribution 3 Gaussian mixtures Arbitrary training initial estimation of transition probabilities

12. humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis Recognition via HMM (Why HMMs?) Stochastic models fit the nature of the gestures Fast convergence due to effective training algorithms Sufficient modeling of the temporal aspect of gestures Continuous HMMs suitable for gesture- level classification

13. humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis HMM overview

14. humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis Recognition via HMM

15. humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis Results Gesture Attention ClappingExplain One hand Oh my godWaveGo away Unclassified Attention 90.9000009.1 0 Clapping 075250000 0 Explain 00600000 40 One hand 00083.3016.70 0 Oh my god 008.3083.300 8.3 Wave 25000062.512.5 0 Go away 2500000750

16. humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis Expressivity features analysis Overall activation Spatial extent Temporal Fluidity Power/Energy Repetitivity

17. humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis Overall activation Considered as the quantity of movement during a conversational turn Computed as the sum of the motion vectors’ norm

18. humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis Spatial extent Modeled by expanding or condensing the entire space in front of the agent that is used for gesturing Calculated as the maximum Euclidean distance of the position of the two hands The average spatial extent is also calculated for normalization reasons

19. humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis Temporal The temporal parameter of the gesture determines the speed of the arm movement of a gesture’s meaning carrying stroke phase and also signifies the duration of movements (e.g., quick versus sustained actions)

20. humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis Fluidity Differentiates smooth/graceful from sudden/jerky ones. This concept seeks to capture the continuity between movements, the arms’ trajectory paths as well as the acceleration and deceleration of the limbs To extract this feature from the input image sequences we calculate the sum of the variance of the norms of the motion vectors

21. humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis Power/Energy The power is actually identical with the first derivative of the motion vectors calculated in the first steps

22. humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis Results of expressivity analysis EF variation Overall Activation Spatial Extent TemporalFluidityPower/Energy

23. humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis Expressive synthesis A system that mimics user’s behaviour through the analysis of facial and gesture signals and expressivity

24. humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis Synthesis Greta Platform BAP calculation Plane assumption Inverse kinematics Manual adaptation Expressivity features variations implemented in Greta’s BAP interpolation

25. humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis Synthesis Results