Matthias Wimmer, Ursula Zucker and Bernd Radig Chair for Image Understanding Computer Science Technische Universität München { wimmerm, zucker, radig Human Capabilities on Video-based Facial Expression Recognition

/10 Technische Universität München Ursula Zucker Motivation  Facial Expression Recognition  goal: human-like man-machine communication  six universal facial expressions [Ekman]: anger, disgust, fear, happiness, sadness, surprise  minimal muscle activity -> reliable recognition is difficult  recognition rate of state-of-the-art approaches: ~ 70%  Question  How reliable do humans specify facial expressions? -> survey to determine human capabilities

/10 Technische Universität München Ursula Zucker The Facial Expression Database Cohn-Kanade AU-Coded Facial Expression Database  488 image sequences (containing 4 up to 66 images)  each showing one of the six universal facial expressions  no natural facial expressions (simulated ground truth)  no context information

/10 Technische Universität München Ursula Zucker Description of Our Survey  Execution of the Survey  participants are shown randomly selected sequences  250 participants  5413 annotations -> approx. 11 per sequence

/10 Technische Universität München Ursula Zucker Evaluation  Evaluation of the Survey  no ground truth -> comparison of the annotations to one another  annotation rate for each sequence and each facial expression  relative agreement for an expression  confusion between facial expressions  Comparison to algorithms  recognition rate

/10 Technische Universität München Ursula Zucker Annotation Rate for Each Sequence  Explanation:  488 rows  1 row = 1 sequence  darker regions denote a higher annotation rate  sorted by similar annotation  Result:  happiness: best annotation rates  surprise and fear: get confused often  fear: difficult to tell apart

/10 Technische Universität München Ursula Zucker Relative Agreement  Explanation:  example: annotating the sequences as happiness ~ 350 sequences annotated as happiness by nobody, ~ 50 sequences annotated as happiness by everybody  well-recognized facial expressions have peaks at “0” and at “1”

/10 Technische Universität München Ursula Zucker Confusion Between Facial Expressions  fear and surprise: high confusion  happiness and disgust: low confusion confusion rate

/10 Technische Universität München Ursula Zucker Comparison: humans vs. algorithms  ground truth: provided by Michel et. al.  Results:  Michel et. al.: worse at recognizing anger  Schweiger et. al.: worse at recognizing disgust, fear, happiness and on the average

/10 Technische Universität München Ursula Zucker Conclusion  Survey applies similar assumptions as algorithms:  consideration of visual information only  no context information  no natural facial expressions  Summary of our results:  poor recognition rate of humans – worse than expected  some facial expressions get confused easily  Conclusion & Outlook:  integration of more sources of information is highly recommended, e. g. audio/language, context,...

/10 Technische Universität München Ursula Zucker Thank you