A Fuzzy System for Emotion Classification based on the MPEG-4 facial definition parameter set Nicolas Tsapatsoulis, Kostas Karpouzis, George Stamou, Fred Piat and Stefanos Kollias Image, Video and Multimedia Systems Laboratory National Technical Univ. of Athens

Problem Statement Describe archetypal emotions using the FAPs of MPEG-4 Approximate FAPs through some Facial protuberant points Combine the emotion wheel of Whissel with a fuzzy inference system to extend to broader variety of emotions

Emotion Analysis: Engineers and Psychological Researchers Engineers concentrated (basicallly) on archetypal emotions -surprise, fear, joy, sadness, disgust, anger. Psychological researchers investigated variety of emotions Their results are not easily implemented Some hints can be obtained Whissel suggests that emotions are points in a two-dimensional space

Whissel’s emotion wheel Axes: activation –evaluation Activation: degree of arousal Evaluation: degree of pleasantness

FAPs and Archetypal Expressions Anger squeeze_l_eyebrow (+) lower_t_midlip (-) raise_l_i_eyebrow (+) close_t_r_eyelid (-) close_b_r_eyelid (-) squeeze_r_eyebrow(+) raise_b_midlip (+) raise_r_i_eyebrow (+) close_t_l_eyelid (-) close_b_l_eyelid (-) Sadness raise_l_i_eyebrow (+) close_t_l_eyelid (+) raise_l_m_eyebrow (-) raise_l_o_eyebrow (-) close_b_l_eyelid (+) raise_r_i_eyebrow (+) close_t_r_eyelid (+) raise_r_m_eyebrow (-) raise_r_o_eyebrow (-) close_b_r_eyelid (+) Surprise raise_l_o_eyebrow (+) raise_l_i_eyebrow (+) raise_l_m_eyebrow (+) squeeze_l_eyebrow (-) open_jaw (+) raise_r_o_eyebrow (+) raise_r_i_eyebrow (+) raise_r_m_eyebrow(+) squeeze_r_eyebrow (-)

Facial animation in MPEG-4 Motion represented by FAPs (Facial Animation Parameters) e.g. raise_l_o_eyebrow, raise_r_i_eyebrow, open_jaw Normalized to standard distances of rigid areas in the face, e.g. left eye to right eye (ES0) or nose to eye level (ENS0)

Synthetic faces in MPEG-4 Defined through FDPs (Face Definition Points)

Emotion Words due to Whissel Activat.Evaluat Activat.Evaluat Afraid4.93.4Angry Bashful 2 2.7Delighted Disgusted 5 3.2Eager55.1 Guilty 4 1.1Joyful Patient Sad Surprised6.55.2

Joy close_t_l_eyelid (+) close_b_l_eyelid (+) stretch_l_cornerlip (+) raise_l_m_eyebrow (+) close_t_r_eyelid (+) close_b_r_eyelid (+) stretch_r_cornerlip (+) raise_r_m_eyebrow(+) lift_l_cheek (+) lower_t_midlip (-) OR open_jaw (+) lift_r_cheek (+) raise_b_midlip (-) Disgust close_t_l_eyelid (+) close_t_r_eyelid (+) lower_t_midlip (-) close_b_l_eyelid (+) close_b_r_eyelid (+) open_jaw (+) squeeze_l_cornerlip (+) AND / OR squeeze_r_cornerlip (+) Fear raise_l_o_eyebrow (+) raise_l_m_eyebrow(+) raise_l_i_eyebrow (+) squeeze_l_eyebrow (+) open_jaw (+) raise_r_o_eyebrow (+) raise_r_m_eyebrow (+) raise_r_I_eyebrow (+) squeeze_r_eyebrow(+) close_t_r_eyelid (-) OR close_t_l_eyelid (-) lower_t_midlip (-) OR lower_t_midlip (+)

Detection of Facial Protuberant Points Automatic detection in images where the face segments are large; semi- automatic procedure otherwise Detection of eyes guides the detection of the other points

“Hierarchical facial features localisation using a morphological approach,” Raphael Villedieu, Technical Report NTUA, June 2000

Features and Linguistic terms Table 4 is used to determine how many and which linguistic terms should be assigned to a particular feature Example: the linguistic terms medium and high are sufficient for the description of feature F 11 Membership functions for feature F4

Fuzzification of the input vector Universe of discourse for the particular features is estimated based on statistics: Example: a reasonable range of variance for F 5 is where m A5, σ A5 and m Su5, σ Su5 are the mean values and standard deviations of feature F 5 corresponding to expressions anger and surprised respectively (see Table 4 before) Unidirectional features like F 11 either the lower or upper limit is fixed to zero

Fuzzy Inference System A 15-tuble feature vector, corresponding to the FAPs depicted in Table 3. Output is an n-tuple, where n is the number of modeled emotions; for the archetypal output values express the degree of the belief that the emotion is anger, sadness, joy, disgust, fear or/and surprise Fuzzification: Use of Table 4 (Estimation of the FAPs range intervals) Fuzzy Rule Base: obtained from psychological studies; use of Whissel’s activation parameter; express the a-priori knowledge of the system. If–then rules are heuristically constructed from Tables 2 and 4

Recognition of a broader variety of emotions Estimate which features participate to the emotions Modify the membership functions of the features to correspond to the new emotions Define six general categories corresponding to archetypal emotions Example: Category fear contains also worry and terror; model these by translating appropriately the positions of the linguistic terms, associated with the particular features, in the universe of discourse axis.

Modifying the membership functions using the activation parameters Let activation values a Y and a X corresponding to emotions Y and X Rule 1: Emotions of the same category involve the same features F i. Rule 2: Let μ ΧZi and μ YZi be the membership functions for the linguistic term Z corresponding to F i and associated with emotions X and Y respectively. If the μ ΧZi is centered at value m XZi of the universe of discourse then μ YZi should be centered at: Rule 3: a Y and a X are known values obtained from Whissel’s study

Experimental Results