1 Emotion Classification Using Massive Examples Extracted from the Web Ryoko Tokuhisa, Kentaro Inui, Yuji Matsumoto Toyota Central R&D Labs/Nara Institute of Science and Technology Coling 2008

2 Emotion Classification Given an input sentence, classify the sentence into 10 emotion classes Need to construct an emotion-provoking corpus (EP corpus) Example: I was disappointed because the shop was closed and I ’ d traveled a long way to get there Emotion: disappointment Event: the shop was closed and I ’ d traveled a long way to get there

3 Building an EP corpus (1/8) Ten emotions: happiness, pleasantness, relief, fear, sadness, disappointment, unpleasantness, loneliness, anxiety, anger Built a hand-crafted lexicon of emotion words from the Japanese Evaluation Expression Dictionary 349 emotion words

4 Building an EP corpus (2/8)

5 Building an EP corpus (3/8) A subordinate clause was extracted as an emotion-provoking event if (a) it was subordinated to a matrix clause headed by an emotion word (b) the relation between the subordinate the matrix clauses is marked by eight connectives (Japanese)

6 Building an EP corpus (4/8) “ It suddenly started raining ” provokes disappointment

7 Building an EP corpus (5/8) Applying the lexical patterns to the Japanese Web corpus, which contains 500 million sentences, 1.3 million events were collected

8 Building an EP corpus (6/8)

9 Building an EP corpus (7/8) An annotator evaluated 2000 randomly chosen events Correct: correct example Context-dep: context-dependent Error: error example

10 Building an EP corpus (8/8)

11 Sentiment Polarity Classification (1/5) Finding neutral events is difficult Collected 1000 sentences randomly from the web and investigate their polarity

12 Sentiment Polarity Classification (2/5) Two-step approach First classify a given input into three sentiment polarity classes, either positive, negative or neutral Then classify only those judged positive or negative into the 10 fine-grained emotion classes

13 Sentiment Polarity Classification (3/5) The sentiment classification model is trained with SVMs Test sentence is neutral if the output of the classification model is near the decision boundary Features are 1-gram, 2-gram and 3- gram extracted from word-polarity lattice

14 Sentiment Polarity Classification (4/5) Features: child, positive, child-of, positive-of, child-of-education, …

15 Sentiment Polarity Classification (5/5) Polarity value of each word is defined in a sentiment dictionary, which includes 1880 positive words and 2490 negative words One annotator identified positive and negative words from the 50 thousand most frequent words samples from the Web

16 Emotion Classification Use k-nearest-neighbor approach (kNN)

17 Experiments: Sentiment polarity classification (1/2) TestSet1: 6 subject speakers produce 31 positive utterances, 34 negative utterances, and 25 neutral utterances TestSet2: Used the 1140 samples that were labeled Correct before 491 positive samples, 649 negative samples Add 501 neutral samples from the Web

18 Experiments: Sentiment polarity classification (2/2)

19 Experiments: Emotion classification (1/3) TestSet1 (2p, best): Two annotators annotated each positive or negative sentence in TestSet1 with exactly one of the 10 emotion classes. A model ’ s answer is correct if it was identical with either of the two labels TestSet1 (1p, acceptable): One annotator is asked to annotate each positive or negative sentence in TestSet1 with all the emotions involved in it. A model ’ s answer was considered correct if it was identical with one of the labeled classes TestSet2: Use the results from the evaluation of EP corpus ’ quality

20 Experiments: Emotion classification (2/3) Baseline Kozareva ’ s PMI unsupervised model Use Google search engine to obtain an emotion and a word ’ s co-occurrence frequency k-NN 1-NN, 3-NN, 10-NN

21 Experiments: Emotion classification (3/3)