RECOGNIZING FACIAL EXPRESSIONS THROUGH TRACKING Salih Burak Gokturk

OVERVIEW PROBLEM DESCRIPTION TRAINING STAGE TESTING STAGE EXPERIMENTS CONCLUSION

Components of the recognition system Analysis -Face Tracking Intelligence -Support Vector Machine Classifier Shape Parameters Training with stereo DataClassifier Testing with mono New Data Output

PROBLEM DESCRIPTION(Tracking ) ?

PROBLEM DESCRIPTION (Recognition) X(t) [ Rigid, Open Mouth, Smile] ? Training DataClassifier Testing New DataOutput

OVERVIEW PROBLEM DESCRIPTION TRAINING STAGE TESTING STAGE EXPERIMENTS CONCLUSION

p - degrees of freedom Stereo Tracking Data Monocular Tracking And Classification Learn Shape

Support Vector Machines (SVM) - Best discriminating hypersurface between two class of objects - Map the data to high dimension using a map function  - The hypersurface in the feature space corresponds to a hyperplane in the mapped space Training DataClassifier Testing (Classifier) New DataOutput

OVERVIEW PROBLEM DESCRIPTION TRAINING STAGE TESTING STAGE EXPERIMENTS CONCLUSION

LUKAS TOMASI KANADE OPTICAL FLOW TRACKER EXTENDED TO 3D X(t) I(x(t)) I(t+1) TIME t+1 ? X(t+1)

One to Many Application of Support Vector Machines (SVM) - One hypersurface per class is calculated - A new data is tested for each hypersurface - A different probability is assigned to ith class

OVERVIEW PROBLEM DESCRIPTION TRAINING STAGE TESTING STAGE EXPERIMENTS CONCLUSION

-Training (Stereo) with 2 people, totally 240 frames - Testing with 3 people - 5 expressions: neutral, open mouth, close mouth, smile, raise eyebrow - velocity term is added to the shape vector: - Two other classifiers were tested: 1 - Clustering 2 – N-Nearest Neighbor

MOVIE (1)

MOVIE (2)

Decision of the system Input  NeutralOpen mouth Close mouth SmileRaise eyebrow Neutral (44) Open mouth (80) Close Mouth (50) Smile (87) Raise Eyebrow (21) Performance of the system for different expressions Table 1

Comparison Between Different Methods SVM with kernel erbf SVM with kernel rbf ClusteringN-Nearest with N=9 N-Nearest with N=5 Same person 176/182170/182161/182173/182 Total256/282253/282242/283255/282253/282 Table 2

-Training (Stereo) with 1 person, totally 130 frames - Testing with 3 people - 5 expressions: neutral, open mouth, close mouth, smile, raise eyebrow Comparison Between Different Methods with only one person training set SVM with kernel erbf SVM with kernel rbf ClusteringN-Nearest with N=9 N-Nearest with N=5 Same person98/11099/110109/ /110 Total216/282207/282233/282231/282229/282 Table 3

-Training (Stereo) with 2 people, totally 240 frames - Testing with 3 people - 3 emotional expressions: neutral, happy, surprise - Transition between expressions are separated Comparison Between Different Methods with three emotional expressions SVM with kernel erbf SVM with kernel rbf Cluster ing N- Nearest with N=9 N-Nearest with N=5 N-Nearest with N=3 N-Nearest with N=1 Same person 164/165165/165152/165163/165164/165 Total222/228223/228213/228225/228224/228223/228 Table 4

Performance Comparison Between Previous Expression Recognition Work Recognition Rate Pose Change Number of Expressions Test/Train Subject Number of Data Comments Chen et.al, ICME 2000 %89Direct camera view 7Different subject 470 images Problem with different people Wang et.al, AFGR 1998 %96Direct camera view 3Different subject 29 image sequence Sequence classification (easier) Lien et.al, AFGR 1998 %85-%93~10 degrees rotation 4Different subject ~130 images Only upper part of the face is classified Hiroshi et.al, ICPR 1996 %70~45-60 degrees rotation 5Same subject 900 images Permits for rotations, but rates are not as good Chang et.al, IJCNN 1999 %92Direct camera view 3Different subject 38 imagesSmall test and training set Matsuno et.al, ICCV 1995 %80Direct camera view 4Different subject 45 imagesSmall test and training set Hong et.al, AFGR 1998 %65-%85Direct camera view 7Same and different subject ~250 images %85 with known person % 65 with unknown person Hong et.al, AFGR 1998 %81-%97Direct camera view 3Same and different subject ~250 images %97 with known person % 81 with unknown person Sakaguchi et.al, ICPR 1996 %84Direct camera view 6Same subject -The test and training set not mentioned Our Work%91~70-80 degrees rotation 5Different subject 282 images Table 2 Our Work%98~70-80 degrees rotation 3Different subject 228 images Table 4 - Emotional Expressions

OVERVIEW PROBLEM DESCRIPTION TRAINING STAGE TESTING STAGE EXPERIMENTS CONCLUSION

Future Work Conclusions - Breakthrough facial expression recognition rates. - 3-D is the right way to go… - Test with more subjects and expressions. - further application to face recognition (?)