1. Matthias Wimmer, Ursula Zucker and Bernd Radig Chair for Image Understanding Computer Science Technische Universität München { wimmerm, zucker, radig }@in.tum.de Human Capabilities on Video-based Facial Expression Recognition

2. 2007-09-10 2/10 Technische Universität München Ursula Zucker Motivation  Facial Expression Recognition  goal: human-like man-machine communication  six universal facial expressions [Ekman]: anger, disgust, fear, happiness, sadness, surprise  minimal muscle activity -> reliable recognition is difficult  recognition rate of state-of-the-art approaches: ~ 70%  Question  How reliable do humans specify facial expressions? -> survey to determine human capabilities

3. 2007-09-10 3/10 Technische Universität München Ursula Zucker The Facial Expression Database Cohn-Kanade AU-Coded Facial Expression Database  488 image sequences (containing 4 up to 66 images)  each showing one of the six universal facial expressions  no natural facial expressions (simulated ground truth)  no context information

4. 2007-09-10 4/10 Technische Universität München Ursula Zucker Description of Our Survey  Execution of the Survey  participants are shown randomly selected sequences  250 participants  5413 annotations -> approx. 11 per sequence

5. 2007-09-10 5/10 Technische Universität München Ursula Zucker Evaluation  Evaluation of the Survey  no ground truth -> comparison of the annotations to one another  annotation rate for each sequence and each facial expression  relative agreement for an expression  confusion between facial expressions  Comparison to algorithms  recognition rate

6. 2007-09-10 6/10 Technische Universität München Ursula Zucker Annotation Rate for Each Sequence  Explanation:  488 rows  1 row = 1 sequence  darker regions denote a higher annotation rate  sorted by similar annotation  Result:  happiness: best annotation rates  surprise and fear: get confused often  fear: difficult to tell apart

7. 2007-09-10 7/10 Technische Universität München Ursula Zucker Relative Agreement  Explanation:  example: annotating the sequences as happiness ~ 350 sequences annotated as happiness by nobody, ~ 50 sequences annotated as happiness by everybody  well-recognized facial expressions have peaks at “0” and at “1”

8. 2007-09-10 8/10 Technische Universität München Ursula Zucker Confusion Between Facial Expressions  fear and surprise: high confusion  happiness and disgust: low confusion confusion rate

9. 2007-09-10 9/10 Technische Universität München Ursula Zucker Comparison: humans vs. algorithms  ground truth: provided by Michel et. al.  Results:  Michel et. al.: worse at recognizing anger  Schweiger et. al.: worse at recognizing disgust, fear, happiness and on the average

10. 2007-09-10 10/10 Technische Universität München Ursula Zucker Conclusion  Survey applies similar assumptions as algorithms:  consideration of visual information only  no context information  no natural facial expressions  Summary of our results:  poor recognition rate of humans – worse than expected  some facial expressions get confused easily  Conclusion & Outlook:  integration of more sources of information is highly recommended, e. g. audio/language, context,...

11. 2007-09-10 11/10 Technische Universität München Ursula Zucker Thank you