FACIAL EMOTION RECOGNITION BY ADAPTIVE PROCESSING OF TREE STRUCTURES Jia-Jun Wong and Siu-Yeung Cho Forensic and Security Lab School of Computer Engineering Nanyang Technological University Singapore presented by

Abstract Introduction –What are the basic emotions? –Existing systems? Facial Emotion Tree Structures –What are Tree Structures? –Why use Tree Structures? –How to process Tree Structures? Performance Evaluation –Under perfect feature location situations –With missing features Conclusions

Introductions There are six basic emotions according to psychologists Emotions are innate and universal Facial Emotions are revealed faster FACS by Paul Ekman –100 hours to train Computerize methods –Surface texture analysis with PCA –Facial Motion through optic flow –ICA, etc.

Facial Emotion Recognition System Emotion Recognition Feature Extraction Face Detection Face Cropping and Resizing Image Processing Eyes Detection Nose Detection Mouth Detection Gabor FilterFeature Extraction Facial Emotion Tree Structure Transformation Probabilistic Based Recursive Neural Network Facial Emotion Tree Structure Representation Recognised Emotion State Feature Locations Localised Gabor Features Capture Image

What are Tree Structures? Traditionally features are stored and used in a flat vector format –Simple to implement and use –This loses feature to feature relationship information Flat feature vector can be transformed into tree structures –Encodes feature to feature relationship information –More flexibility in recognition A B C D c a bd f e Scene Sky House Ground AB C abc de f

Face Emotion Tree Structure (FEETS) L3 L1 L2 L4 L5

Adaptive Processing of Tree Structures Step 1: Encode Data into Tree Structure Step 2: Feed each node into a interconnecting Neural Node F00 F09 F18 F01 F02 Neural Node F02 F01 F09 F18 F00

Adaptive Processing of Tree Structures Maximum number of children for a node, which is the branch factor is assumed for a task. Tree Node GMM_G GMM_1 Output, y Input attributes, uChildren’s output, y Output layer Hidden layer Input layer

Probabilistic Recursive Model Class likelihood function Unsupervised Learning –Expectation Step –Maximisation Step Supervised Learning –Levenberg Marquardt Algorithm

Features Used Localized Gabor Features Biological relevance and computational properties. Captures the properties of –spatial localization, –quadrature phase relationship.

Feature Locations Four primary feature locations –the center of the left eye, –center of the right eye, –tip of the nose, –the center of the lips. 60 Extended Features

Performance Evaluations Database used –Japanese Female Facial Expression (JAFFE) Database 213 images of 7 facial expressions (including neutral) posed by 10 Japanese female models SubjectsTrainingTest Known14370 Unknown17043

FEETS vs Quadtree FEETS are smaller in size Higher recognition rate

FEETS vs Others

Missing Features Performance Perfect Condition Eyes Missing Nose & Mouth Misssing

FEETS vs Others Database Used CMU Emotion Database

FEETS vs Others

Conclusions FacE Emotion Tree Structures (FEETS) has achieve high recognition rates Robust recognition when there are missing features Smaller footprint than Quadtrees

Thank You