1. AUTOMATIC SPEECH CLASSIFICATION TO FIVE EMOTIONAL STATES BASED ON GENDER INFORMATION ABSTRACT We report on the statistics of global prosodic features of certain emotional speech styles for each gender separately. The components of the investigation: 500 emotionally expressed speech segments (2 male, 2 female actors) 5 basic emotions, i.e., Anger, Happy, Neutral, Sad and Surprise a total of 87 global statistics of energy, pitch and formants Bayes classifier where class pdfs are approximated via Parzen windows or modeled as Gaussians Dimitrios Ververidis and Constantine Kotropoulos ARISTOTLE UNIVERSITY OF THESSALONIKI, DEPARTMENT OF INFORMATICS Box 451, Thessaloniki 540 06, GREECE, e-mail: {jimver, costas}@zeus.csd.auth.gr URL: http://poseidon.csd.auth.gr INTRODUCTION Data FEATURE EXTRACTION EVALUATION OF SINGLE FEATURES ON EACH GENDER SEPARETELY Methods that can classify the emotions in speech would be of great usefulness in computer science, linguistic sciences, psychology, and medical sciences. For example: 1.Automatic Speech Recognition optimization. 2.Parkinson ‘s disease. 3.Improvement of the quality of an interface by detecting the frustration and the dissatisfaction of a user. Public domain Danish Emotional Speech database (DES), obtained after request to Inger Samsø Engberg at the faculty of Institute of Electronic Systems at Aalborg Univ., Denmark. The data used in the experiments are 500 sentences and words that are located between two silent segments, equally separated into two gender categories. Four professional actors, two male and two female are speaking in 5 emotional states, such as anger, happiness, neutral, sadness, and surprise. Global statistical feature estimation: 87 statistics of the pitch, energy, and formant contours are extracted. The statistics are calculated on rising, falling slopes, and maximum/minimum plateaux of the contours. Typical examples are: the maximum, the minimum, the median, the mean, and interquartile range. In order to study the classification ability of each feature, a rating method has been implemented. Each feature is evaluated by the ratio between the between-class variance (σ b 2 ) and the within-class variance (σ w 2 ): L: number of classes : A priori probability of ω i : Random Vector M i : Expected vector of ω i M 0 : Expected vector of the mixture density Pitch features: 20.Mean range 50.Mean value of falling slopes 43.Mean value of rising slopes 18. Maximum value of pitch 22.Interquartile range of pitch 26.Mean value of plateux at minima Energy features: 54.Maximum value 78.Mean value of rising slopes 85.Mean value of falling slopes 86. Median value of falling slopes 79. Median value of rising slopes

2. Bayes classifier with Gauss pdfs BothMalesFemales Correct classification rate 51.6%61.1%57.1% Correct classification rate Human Perception Random 67%20% The Sequential Forward Selection (SFS) algorithm is used for automatic feature selection. The criterion employed is the correct classification rate achieved by the selected features. The correct classification rate is calculated by cross-validation where 90% of the data were used for training and 10% for validation. Visualization in 2D CONFUSION MATRICES StimuliResponse (%) NeutralSurpriseHappinessSadnessAnger Neutral 51 152284 Surprise5647914 Happiness924361318 Sadness1762705 Anger1219261231 StimuliResponse (%) NeutralSurpriseHappinessSadnessAnger Neutral60.82.60.131.74.8 Surprise1059.128.711.3 Happiness8.329.856.41.73.8 Sadness12.61.80.185.20.3 Anger10.28.54.51.775.1 Correct classification rates by humans at 67% Confusion matrices Black color: Humans reach the level of 67% correct classification score. Green color: Bayes classifier with Gaussian pdfs using features 76, 18, 44, 27, 7: When the crossvalidation method was used, a correct classification rate of 51.6% is obtained. (54%, when all data are used for training and testing). Blue color: The aforementioned method used only for male patterns. The method selects different features and achieves a correct classification score of 61.1%. Red color: The aforementioned method used only for female patterns. A correct classification score of 57.1% is achieved. AUTOMATIC FEATURE SELECTION Features selected by SFS with a Bayes classifier when class pdfs are modeled as Gaussian pdfs as a criterion. PCA was used in order to reduce the dimensionality from five dimensions (5D) to two dimensions (2D). Only the samples which belong to the interquartile range of the pdf for each class are shown. The ellipses denote the 60% likelihood contours for a 2-D Gauss model. StimuliResponse (%) NeutralSurpriseHappinessSadnessAnger Neutral6738201 Surprise36018613 Happiness181343621 Sadness1132803 Anger61913656 StimuliResponse (%) NeutralSurpriseHappinessSadnessAnger Neutral 55 136206 Surprise12611467 Happiness121154418 Sadness13445821 Anger61018957 CONCLUSIONS The rates reported can be further improved by analyzing the properties of two­class problems. The features which can separate two classes could be different from those which separate 5 classes. By designing proper decision fusion algorithms, we may combine several two­class classifiers and the overall system could outperform the rates obtained by the five­class classifiers. If the words in the training set could be linguistically different as those in the testing set then the classification would be linguistic unbiased. The error estimation can be done according to the elements outside the diagonal line of the confusion matrix, so that the Bayes classifier matches the human classification and misclassification rates. Classification rates of a Bayes classifier for male subjects at 61.1% Classification rates of a Bayes classifier for female subjects at 57.1% Classification rates of a Bayes classifier for both genders at 51.6% Classifier Step12345678910 Bayes with Gaussian class pdfs (male) 54437481217886918- Bayes with Parzen windows (male) 5474206786581730-- Bayes with Gaussian class pdfs (female) 4378202510776178245 Bayes with Parzen windows (female) 4380183986----- Bayes with Gaussian class pdfs 781845267----- 77% correct classification score for Surprise vs. Happy using features 78, 43, 22, 26, 85. Much classification information is lost because PCA is a lossy transformation. The correct classification scores of the diagonal elements of the four tables, Human classification, Bayes classifier for males only, females only, both genders.