1. An Infant Facial Expression Recognition System Based on Moment Feature Extraction C. Y. Fang, H. W. Lin, S. W. Chen Department of Computer Science and Information Engineering National Taiwan Normal University Taipei, Taiwan

2. 1 Outline Introduction System Flowchart Infant Face Detection Feature Extraction Correlation Coefficient Calculation Infant Facial Expression Classification Experimental Results Conclusions and Future Work

3. 2 Introduction Infants can not protect themselves generally. Vision-based surveillance systems can be used for infant care. Warn the baby sitter Avoid dangerous situations This paper presents a vision-based infant facial expression recognition system for infant safety surveillance. camera

4. 3 The classes of infant expressions Five infant facial expressions: crying, gazing, laughing, yawning and vomiting Three poses of the infant head: front, turn left and turn right Total classes: 15 classes crying gazing laughing yawning vomiting front turn right turn left

5. 4 System Flowchart Infant face detection: to remove the noises and to reduce the effects of lights and shadows to segment the image based on the skin color information Feature extraction: to extract three types of moments as features, including Hu moments, R moments, and Zernike moments Feature correlation calculation: to calculate the correlation coefficients between two moments of the same type for each 15-frame sequence Classification: to construct the decision trees to classify the infant facial expressions

6. 5 Infant Face Detection Lighting compensation To make the skin color detection correctly Infant face extraction Step1: Skin color detection Using three bands S of HSI Cb of YCrCb U of LUX Step2: Noise reduction Using 10x10 median filter Step3: Infant face identification Using temporal information Lighting compensation Skin color detection Noise reduction

7. 6 Infant Face Detection Step 3: Infant face identification

8. 7 Moments To calculate three types of moments Hu moment [Hu1962] R moment [Liu2008] Zernike moment [Zhi2008] Given an image I and let f be an image function. The digital (p, q)th moment of I is given by The central (p, q)th moments of I can be defined as where and The normalized central moments of I where

9. 8 Hu Moment Hu moments are translation, scale, and rotation invariant. normalized central moments

10. 9 Example: Hu Moments crying

11. 10 Example: Hu Moments yawning

12. 11 Example: Hu Moments yawning crying If the infant facial expressions are different then the values of Hu moments are also different.

13. 12 R Moment Liu (2008) proposed ten R moments which can improve the scale invariability of Hu moments. Hu moments

14. 13 Example: R Moments crying Hu moments R moments and Hu moments may have different properties.

15. 14 Zernike Moment Zernike moments of order p with repetition q for an image function f is where To simplify the index, we use Z 1, Z 2,…, Z 10 to represent Z 80, Z 82,…, Z 99, respectively. real part imaginary part

16. 15 Zernike Example: Zernike Moments crying

17. 16 Correlation Coefficients A facial expression is a sequential change of the values of the moments. The correlation coefficients of two moments may be used to represent the facial expressions. Let A i =, i = 1, 2,…, m, indicates the ith moment A i of the frame I k, k = 1, 2,…, n. The correlation coefficients between A i and A j can be defined as where and : the mean of the elements in A i

18. 17 Correlation Coefficients The correlation coefficients between seven Hu moment sequences. H1H1 H2H2 H3H3 H4H4 H5H5 H6H6 H7H7 H1H1 10.87780.9481-0.033-0.571-0.80520.8907 H2H2 10.94740.1887-0.4389-0.87490.9241 H3H3 10.1410-0.6336-0.90440.9719 H4H4 10.0568-0.34310.2995 H5H5 10.7138-0.6869 H6H6 1-0.9727 H7H7 1 yawning

19. Decision Tree Decision trees are used to classify the infant facial expressions. 18 H1H2H1H2 H1H3H1H3 H2H3H2H3 -++ ++- +++ ++- -++ --+ --- +-- --- +-+ H 1 H 3 >0 correlation coefficients

20. 19 Decision Tree The correlation coefficients between two attributes A i and A j are used to split the training instances. Let the training instances in S be split into two subsets S 1 and S 2 by the correlation coefficient, then the measure function is The best correlation coefficient selected by the system is

21. 20 Decision tree construction Step 1: Initially, put all the training instances into the root S R, regard S R as an internal decision node and input S R into a decision node queue. Step 2: Select an internal decision node S from the decision node queue calculate the entropy of node S. If the entropy of node S larger then a threshold T s, then goto Step 3, else label node S as a leaf node, goto Step 4. Step 3: Find the best correlation coefficient to split the training instances in node S. Split the training instances in S into two nodes S 1 and S 2 by correlation coefficients and add S 1, S 2 into the decision node queue. Goto Step 2. Step 4: If the queue is not empty, then goto Step 2, else stop the algorithm.

22. 21 Experimental Results Training: 59 sequences Testing: 30 sequences Five infant facial expressions: crying, laughing, dazing, yawning, vomiting Three different poses of infant head: front, turn left, and turn right Fifteen classes are classified. cryinglaughingdazingyawningvomiting Turn left Front Turn right

23. Feature type: Hu moments Internal nodes: 16 Leaf nodes: 17 Height: 8

24. Experimental Results Testing sequencesClassification results 23 laughing dazing vomiting The classification results of the Hu-moment decision tree

25. 24

26. 25 Experimental Results Testing sequencesClassification results vomiting dazing yawning dazing The classification results of the R-moment decision tree

28. 27 Experimental Results Testing sequencesClassification results crying vomiting crying The classification results of the Zernike-moment decision tree

29. 28 Conclusions The comparison of the results The correlation coefficients of the moments are useful attributes to classify the infant facial expressions. The classification tree created by the Hu moments has less height and number of node, but higher classification rate. Height of the decision tree Number of nodes Number of training sequences Number of testing sequences Classification Rate Hu moments 816+17593090% R moments 1015+17593080% Zernike moments 719+20593087%

30. 29 Conclusions and Future Work Conclusion A vision-based infant facial expression recognition system Infant face detection Moment features extraction Correlation coefficient calculation Decision tree classification Future work To collect more experimental data To fuzzify the decision tree Binary decision trees may have less noise tolerant ability. If the correlation coefficients are close to zero, the noises will greatly affect the classification results.

31. 30