1. TAUCHI – Tampere Unit for Computer-Human Interaction Automated recognition of facial expressi ns and identity 2003 UCIT Progress Report Ioulia Guizatdinova Research Group for Emotions, Sociality, and Computing University of Tampere 10.01.2004

2. TAUCHI – Tampere Unit for Computer-Human Interaction Contents Research problems Aims and Tasks Facial landmark extraction : Methods Facial landmark extraction : Results Future Steps

3. TAUCHI – Tampere Unit for Computer-Human Interaction Research Problems “Automated recognition of facial expressions and identity” Face identification Recognition of facial expressions

4. TAUCHI – Tampere Unit for Computer-Human Interaction Face classes 1…N1…N Person A Person Z … … …… Face database Input face Face identification Unrecognized face still image/video signal Recognition system Classification of input face to one of existing face classes stored in database Rejection of input face as unrecognized/unknown face classification rejection Research Problems

5. TAUCHI – Tampere Unit for Computer-Human Interaction Facial expressions affect face recognition because a variability of facial landmarks in their appearance is high Humans are good in recognizing facial identity regardless of changes in facial expressions Computer-aided systems of face recognition are dramatically compromised by changes in facial expressions Recognition of facial expressions Research Problems

6. TAUCHI – Tampere Unit for Computer-Human Interaction Aim of research The primary aim of this research is to do theoretical and experimental investigation on the possibilities to automatically recognize facial identity independent of changes in facial expressions For that purpose two 2D recognition systems will be developed -Recognition system of facial expressions -Expression-invariant system of facial identity recognition

7. TAUCHI – Tampere Unit for Computer-Human Interaction Tasks Extraction and classification of facial landmarks, namely, regions of eyes/eye-brows, nose, and mouth from still images Detection and recognition of facial expressions - how facial muscle activations can change appearance of a face during emotional reactions? Expression-invariant recognition of facial identity

8. TAUCHI – Tampere Unit for Computer-Human Interaction Four regions of interest have been selected as most informative for further recognition steps –right eye-brow / eye –left eye-brow / eye –nose –mouth Methods of landmark extraction Uses knowledge on geometrical structure of human faces Based on geometrical features of facial landmarks, such as position of eyes/eye-brows, nose, and mouth Feature-b ased method

9. TAUCHI – Tampere Unit for Computer-Human Interaction Methods of landmark extraction Template-b ase method Represents a face as a feature map/template of original facial image Local oriented edges are used to construct a feature map of the facial image Orientation of edges has been determined with step of 22.5  and encoded as 0,1,…..15 0 1 2 15 3 4 5 6 7 8 9 14 13 12 11 10 22.5 ° Oriented edges extracted in left eye region

10. TAUCHI – Tampere Unit for Computer-Human Interaction Methods of landmark extraction Database Tests were performed using Pictures of Facial Affect [1] 110 images with 7 basic facial displays: happiness, surprise, fear, anger, disgust and neutral expression Images were first normalized to three pre-set sizes 100X150, 200X300 and 300X400 in order to test the effect of image size to the operation of the algorithms In sum 110 x 3 = 330 images were used for algorithm testing [1] Ekman, P., Friesen, W. V., & Hager, J.C. (2002) Facial Action Coding System (FACS). Published by A Human Face, Salt Lake City, UTAH: USA

11. TAUCHI – Tampere Unit for Computer-Human Interaction Input face (RGB) Normalization (grey-level scale) Different resolution levels Resolution level 0 Resolution level 2 Transformation Pre-processing algorithms RGB – grey-level transformation Multiresolution image representation was performed using a recursive Gauss transformation Facial landmark extraction

12. TAUCHI – Tampere Unit for Computer-Human Interaction Final feature map Final feature map has been constructed on base of local oriented edges extracted in each point of grey-level image at each resolution level with exception of points which had the contrast values less than threshold Extraction of local edges has been performed by calculation of difference between two oriented Gaussians with shifted kernels, which allows determining both orientation and contrast of local edge Map of detected points of interest Points of interest have been grouped - if the distance between points of interest was less than the threshold the points were grouped - otherwise ignored Facial landmark extraction

13. TAUCHI – Tampere Unit for Computer-Human Interaction matching Edge orientation Number of points of interest Right Eye Left Eye Nose Mouth Orientation portraits of the facial landmarks Detected regions of interest have been compared with orientation portraits of facial landmarks I have constructed earlier Regions which did not correspond to the portraits have been ignored Pre-knowledge about facial structure have been used Facial landmark extraction

14. TAUCHI – Tampere Unit for Computer-Human Interaction Finally, facial landmarks have been detected! Examples of feature maps of high-contrast oriented edges detected from the expressive images Facial landmark extraction neutraldisgustfear

15. TAUCHI – Tampere Unit for Computer-Human Interaction Performance of the facial landmark detection algorithm averaged by all expressions for three image sizes Results

16. TAUCHI – Tampere Unit for Computer-Human Interaction Performance of the facial landmark detection algorithm averaged by all expressions for three image sizes Results

17. TAUCHI – Tampere Unit for Computer-Human Interaction Right Eye Left Eye NoseMouth Performance of the feature detection system for three image sizes. N-neutral; H-happiness; Sd-sadness; F-fear; A-anger; Sr-surprise; D-disgust (a) (c)(d) (b) Results

18. TAUCHI – Tampere Unit for Computer-Human Interaction Right Eye Left Eye NoseMouth Performance of the feature detection system for three image sizes. N-neutral; H-happiness; Sd-sadness; F-fear; A-anger; Sr-surprise; D-disgust (a) (c)(d) (b) Results

19. TAUCHI – Tampere Unit for Computer-Human Interaction Algorithms are slow – about few seconds Errors in groupping points of interest (red rectangles a, b, c) Some landmarks are undetectable (d) Results Problems (a)(b)(c)(d)

20. TAUCHI – Tampere Unit for Computer-Human Interaction To improve detection of nose and mouth regions two alternatives are proposed -The first one is selection of different thresholds for detection and groupping of points of interest for different resolution levels -The second alternative requires more careful processing of detected regions and searching different landmark parts such as eye and mouth corners and nostrils. Results Recommendations

21. TAUCHI – Tampere Unit for Computer-Human Interaction Future steps Full article about automated expression-invariant detection of facial landmarks; short article about how emotions affect cognitive functioning and how this knowledge might be implicated for HCI To improve landmark detection; implement prototype of 2D recognition system of facial expressions (iExpRec) To implement and test iExpRec To implement of expression-invariant 2D facial identity recognition system (iFaceRec).

22. TAUCHI – Tampere Unit for Computer-Human Interaction Thank you f r your attention!