Learning Subjective Adjectives From Corpora Janyce M. Wiebe New Mexico State University Office of Naval Research grant N00014-95-1-0776.

Introduction Learning evaluation and opinion clues Distributional similarity process Refinement with lexical features Improved results from both Work involves learning clues of evaluation and opinion in text Corpus-based work, meaning these clues are learned from bodies of text. I use the results of a distributional similarity process, seeded by a small amount of human annotation And then refine those features with lexical semantic features which can also be learned from corpora Found that features based on their intersection have higher precision than features based on each alone

Banfield 1985, Fludernik 1993, Wiebe 1994 Subjectivity Tagging Recognizing opinions and evaluations (Subjective sentences) as opposed to material objectively presented as true (Objective sentences) Part of a larger project investigating subjectivity, which involves aspects of language used to express opinions and evaluations. Subjectivity tagging is distinguishing sentences used to present opinions and evaluations from sentences used to objectively present factual material.. For pointers to work on subjectivity, you can look at these citations in the paper. Banfield 1985, Fludernik 1993, Wiebe 1994

Examples At several different levels, it’s a fascinating tale. subjective Bell Industries Inc. increased its quarterly to 10 cents from 7 cents a share. objective Example of a simple subjective sentence Example of a simple objective sentence

Types ? “Enthused” “Wonderful!” “Great product” “Complained” “Speculated” “Maybe” “Complained” “You Idiot!” “Terrible product” This work focuses mainly on three types of subjectivity. Negative evaluation, positive evaluation, and speculation Clear examples of negative subjectivity: Why the term subjectivity? Which is due to the linguist Ann Banfield. The intuition is …

Subjectivity ? “Enthused” “Wonderful!” “Great product” “Complained” “You Idiot!” “Terrible product” “Speculated” “Maybe” Need to attribute the evaluation, emotion, etc. to someone A “subject”. The person whose evaluation, emotion, uncertainty is being expressed. Many different varieties within the subtypes – showing emotion for the sake of the pictures. If they pursue: Can tease things apart most clearly in novels. Objective narration. Also reporters and people who purport to objectivity. Like any aspect of language, clear prototypical instances, and lots of borderline Areas. Interesting to explore where people agree and don’t, which we have and are in the future doing.

Subjectivity Same word, different types “Great majority” objective “Great!“ positive evaluative “Just great.” negative evaluative Lots of complexities which make subjectivity interesting and non-trivial. For one thing, context is important. The word great, for example, can be used in an objective expressions, as in the the great majority of seagulls can fly. Obviously, it can be used positively. But it can also be used negatively, as in great.

“It’s the best!”, he gushed. Subjectivity Multiple types, sources, targets “It’s the best!”, he gushed. - + As well, there might be multiple types, sources, and targets of subjectivity. The speaker “he” is expressing positive evaluation, but simultaneously, by using “gushed” instead of something neutral like “said”, the writer could be expressing negative subjectivity. Writer He It

Applications: Flame Recognition R [ 13: Julia ] Re: BILL WARRINER!!!! R [ 19: Suzanne ] Re: BILL WARRINER!!!! RS < 16: Suzanne > Re: BILL WARRINER!!!! R [ 26: Doug Bone & Jacqui D] Re: A bin full of buds R [ 24: Karin Adamczyk ] Rose hips? R [ 88: Colette Tremblay ] Re: Rose hips? (long) R [ 8: Karin Adamczyk ] R [ 29: Kay Cangemi ] Re: Rose hips? R [ 23: Karin Adamczyk ] R [ 30: Karin Adamczyk ] R [ 18: BCD ] Re: red(as in wine red) roses R [ 32: Laura Johnson-Kelly ] RS [ 3: PattReck ] R [ 27: Bugman ] Re: BILL WARRINER!!!! R [ 37: Bill ] R < 41: Celeste > Subjectivity processing could be used in many text-processing Flames in newsgroups and listservs are characterzied by strong negative Evaluation directed toward another poster. The goal would be to alert the Reader to the presence of flames.

Review Mining From: Hoodoo>hoodooBUGZAPPER@newnorth.net> Newsgroups: rec.gardens Subject: Re: Garden software I bought a copy of Garden Encyclopedia from Sierra. Well worth the time and money. Filter through newsgroups and listserv discussions for reviews of products -- information retrieval of reviews. For example, here is a positive Review of a software product.

Information Extraction Northwest Airlines settled the remaining lawsuits, a federal judge said. objective “The cost of health care is eroding our standard of living and sapping industrial strength,” complains Maher. subjective Here are two similar sentences, in that both are speech sentences, But the first is objective while the second is subjective. Here, this is being presented as factual, with a federal judge the source of information, And it could be treated that way by an information extraction system. But this sentence is merely presenting a complaint – we don’t want to treat This as a fact, with the complainer a source of information.

Other Applications Clustering documents by ideology Text summarization Style in machine translation and generation There are other applications for which subjectivity tagging could be exploited.

Overview Identify large set of candidate clues Existing resources are not sufficient Not consistently marked for subjectivity Not customized to the genre Learn lexical clues from corpora In this phase of the project, we are focusing on identifying Candidate clues, words that have common subjectivity Uses like great. Our idea is to generate a large set of candidates, To be customized later for particular uses. Existing lexical resources are useful, and we plan to use them, but They aren’t subjective. Dictionaries are not consistently marked For subjectivity. As well, they aren’t customized to the genre. We are interested in applications in newsgroups and listservs and have done some subjectivity annotation in those genres. There is a wide variety and creativity in subjectivity clues in those genres. So, this project is to learn clues from corpora, or bodies of text.

Wiebe et al. 1999; Bruce & Wiebe 1999 Corpus and Annotation Subjectivity tags assigned by multiple annotators to 1001 WSJ sentences Tags representing consensus opinions obtained with EM The data used in this work is a set of wall street journal sentences. In previous work, multiple humans classified the sentences as either subjective or objective. There was a round of discussion and revision of the coding manual, and Then The humans reached good agreement. Details are in this paper. The tags used in the current work consensus opinions of the judges. They Were obtained using the expectation maximization algorithm as described in this paper. We found in previous work (point to the second paper) that simply whether an adjective appears in the sentence is useful for recognizing subjective sentences. So, in this paper, we take the performance of a simple adjective feature as a baseline, and work on identifying higher-quality adjective features for subjectivity tagging. Wiebe et al. 1999; Bruce & Wiebe 1999

Adjectives Classifications correlated with adjectives Adjectives extracted from annotations They promised [e+ 2 yet] more for [e+ 3 really good] [e? 1 stuff]. "It's [e? 3 really] [e- 3 bizarre]," says Albert Lerman, creative director at the Wells agency. More detailed manual annotations were performed for this work. The annotators were asked to identify the expressions that made the Subjective sentences subjective. They indicated strength and type of subjectivity. Go through the sentences. Since we are focusing on adjectives, we extracted the adjectives from the most strongly rated expressions of one annotator. These were used as seeds to the distributional similarity process I’ll talk about next. If a related question: we have plans to take advantage of these contextual influences as in word-sense disambiguation. Here we are demonstrating a process for finding large candidate classes.

Lin’s Distributional Similarity have a brown dog R1 R3 R2 R4 Word R W I R1 have have R2 dog brown R3 dog . . . There are a number of methods for distributional similarity that have been used in NLP. I used Dekang Lin’s work. The general idea is to assess word similarity based on the distributional pattern of words. First, the corpus Is parsed with Lin’s dependency grammar, and these dependencies Triples are extracted. (run your finger both ways over the arrows). Lin’s work was good for me because he processes many different kinds of Relations, including those involving adjectives. Lin 1998

Lin’s Distributional Similarity Word1 Word2 R W R W R W R W R W R W To assess the similarity between word1 and word2, Look at the words correlated with Word1; the words correlated with Word2. Intuitively, the more overlap there is between these sets, the more similar the words are judged to be. The similarity metric takes into account not only the size of the set, but also the mutual information between each word and the members of the intersection set. For reference: More precisely, sum the mutual information measures of each word with members of the intersection And divide by the sum of the mutual information measures of the word And it’s correlated words.

Bizarre strange similar scary unusual fascinating interesting curious tragic different contradictory peculiar silly sad absurd poignant crazy funny comic compelling odd Often, distributional similarity is used to aid syntactic processing and for Semantic similarity . But many people have pointed out that words with distributional similarity need not Be synonyms. For example, nurse and doctor are not synonyms, but they are often used in similar contexts. So I was interested in the idea that the process would find words that are Similar to their evaluative or speculative uses, even though they are not Strictly synonymous. So I used the adjectives extracted from the detailed annotations as seeds, an Extracted up to 20 most similar words according to Lin’s results. Just to look at a couple examples: Here are the words for bizarre. For reference: L. Lee: p(V|N); uses it for syntax, smoothing when data is sparse.

Bizarre strange similar scary unusual fascinating interesting curious tragic different contradictory peculiar silly sad absurd poignant crazy funny comic compelling odd Pretty close synonyms – difference in degree. Up to 20.

Bizarre strange similar scary unusual fascinating interesting curious tragic different contradictory peculiar silly sad absurd poignant crazy funny comic compelling odd Some that are not synonyms, but are similar in that they are evaluative (Note to me: these are not antonyms)

Bizarre strange similar scary unusual fascinating interesting curious tragic different contradictory peculiar silly sad absurd poignant crazy funny comic compelling odd Some we probably want to weed out. In most contexts, only mildly evaluative and may easily appear in objective sentences.

Good bad better best nice poor terrific great decent lousy dismal excellent positive exciting fantastic marvelous strong important dumb fair healthy Lots of pretty close synonyms.

Good bad better best nice poor terrific great decent lousy dismal excellent positive exciting fantastic marvelous strong important dumb fair healthy For these we see similarly evaluative words that are antonyms Here we see antonyms/opposites – which are evaluative in the Opposite direction. Probably good clues of subjectivity.

Experiments 9 10 So, the idea is to seed the distributional similarity clustering process with adjectives extracted from subjective expressions. To test the process multiple times, I did 10-fold cross validation. Viewing this is a mostly supervised process – idea is to seed the clustering process with information extracted from a small amount of detailed manual annotation. So, I used 1/10 training and 9/10.

Experiments Distributional similarity Separate corpus Seeds Seeds + Words 9 10 To test the process multiple times, I did 10-fold cross validation. Viewing this is a mostly supervised process – idea is to seed the clustering process with information extracted from a small amount of detailed manual annotation. So, I used 1/10 training and 9/10. On each fold, adjectives were extracted from the subjective elements And the top 20 most similar words were identified, as I mentioned above.

Experiments Distributional similarity Separate corpus Seeds Seeds + Words Filtering 9 10 S Words p(subjective | s) Some simple filtering was performed, based Only on the training set. Words were removed for which the results are not better than the baseline adjective feature. The final set was evaluated on the test set – the entire corpus minus the 1/10 of the data from which the seeds were selected. P(sentence is subjective | one of these words appears) is on average 7.5% points higher than adjective, and 17.7% points than baseline of choosing the most frequent class. S > Adj > Majority

Lexical features Polarity and Gradability Learned from corpora Statistical processing informed by linguistic insights Different data sets used Then certain lexical semantic features of adjectives were considered, Namely polarity and gradability. Different training and testing sets were used than the corpus I used in this work.

Hatzivassiloglou & Wiebe 2000 Gradability Norm Large Larger Largest * More additional * Very additional Linguistic feature of adjectives – gradable or non-gradable. A gradable adjective participates in comparative constructions And accepts modifying expressions that act as intensifiers or diminishers. Gradable adjectives express properties in varying degrees of strength, Relative to a norm. Likely to be good predictors of subjectivity, given The judgement this involves. In contrast, the non-gradable adjective “additional” cannot be used in this way: for additional, there is no inflected comparative form, and *more additional, and does not involve comparison relative to a norm. Things are not all or nothing. Non-gradable adjectives can appear Modified by grading words, and gradable adjectives can appear non-inflected and not modified by intensifiers. SO, A statistical model was developed to classify adjectives as gradable Or non-gradable taking into account the number of times in a corpus an adjective Appears in a comparative or superlative form, and the number of times It is modified by an intensifier or diminisher like “very”. Question they may ask: these are words that have main gradable/non-gradable usages. It is not that all instances of a word are as a gradable (or non-gradable) Adjective. Example: “more suburban” or “very dead”. Hatzivassiloglou & Wiebe 2000

Hatzivassiloglou & McKeown 1997 Polarity - polarity + polarity object beautiful ugly Corrupt and brutal * Corrupt but brutal Semantic orientation is connected with judgements, it appeared to be a Promising feature for subjectivity. Exploited Linguistic observations: such as that adjectives of the same orientation Are conjoined by conjunctions like and but not but. Developed a statistical method for clustering adjectives into Positive and negative polarity adjectives. The adjectives had been identified to have orientation. Gradability learning algorithm applied to all adjectives in the corpus. A future experiment we plan is to run the methods in sequence – first Gradability, then cluster those into postive and negative polarity. The good results with the intersections of those sets – see next – make this promising. Hatzivassiloglou & McKeown 1997

Experiments Separate corpus Distributional similarity Seeds Seeds + Words Filtering 9 10 This is the process we saw before, with the final result of filtering called S

Experiments Separate corpus Distributional similarity Seeds Seeds + Words Filtering 9 10 For each of the lexical classes, which were learned from a separate corpus, they are intersected with the earlier set S, and the Precision of the resulting set is measured on the test set. Lexical Classification Separate corpus

Results Lex Seed Lex Pol+ + 4.6 +10.8 Pol- +18.5 +18.7 Grad+ + 6.5 +15.3 Shows the difference in precision from the baseline Adjective feature. Seed +7.5

Results Lex Seed Lex Pol+,Grad+ + 6.4 +18.0 Pol-, Grad+ +19.9 +21.4 Shows difference from the baseline adj feature for each Seed +7.5

Future Work Apply process to Netnews and Listservs Apply word-sense disambiguation techniques to potentially subjective expressions Flame recognition and review mining Rerunning the gradability and distributional similarity processes with Lin and Hatzivasiloglou on netnews and listserv data. Such automatic techniques probably will pay off even more in that genre, because probably less coverage of evaluative/insulting terms in standard dictionaries. People are very creative in their use of subjectivity in newsgroups and listservs. disambiguation Question: why not just directly use the linguistic indicators of orientation and gradability? That’s something to experiment with. Note that statistical models were developed to exploit the linguistic indicators; in the case of polarity, a clustering algorithm was used to cluster the adjectives into positive and negative polarity adjectives. Whether you want to do this as a preprocessing step as we did or part of the main classifier probably depends on the learning method used.

Conclusions Learning linguistic knowledge from corpora for a pragmatic task Linguistic information Manual annotation Linguistic constraints Processes improve each other Learning linguistic knowledge from corpora is an active area of NLP research. Much of it focuses on syntactic and semantic information; this work is a case study of learning knowledge useful for a pragamatic task, subjectivity tagging. Statistic processing, but using linguistic information. Used linguistic annotation and insights throughout to support/seed statistical processing. Linguistic information informs statistical processing Linguistic constraints used to learn lexical features Detailed annotations seed clustering process

Application 1: Flame recognition From: pattreck@aol.com (PattReck) Newsgroups: rec.gardens.roses Subject: Re: red(as in wine red) roses My two favorite old reds: Cramoisi Superieure, especially great climbing, and Francis Dubreuil. Also Prospero does well in southern California - aren't you on the west coast? -- Candace

Flames (continued) From: Suzanne <Suzanne_member@newsguy.com> Newsgroups: rec.gardens.roses Subject: Re: BILL WARRINER!!!! >>Wow. You guys are really working poor Suzanne over. >po thang. I thank she been workin over her bottle of Kahlua. ***Up &^%$!!! I've been working at a *job* - no Kahlua! You are a snow-snorting dust-bowl dweller, the dustiest of the dusties. Bill Bradley has the support of the "environmentalists" ha ha ha!

Likely likely possible willing probable receptive unlikely able logical rumored potential counterproductive moot significant hesitant worthy unwilling probably desirable weak forthcoming imminent Noise in our data, because these tended to get filtered out, Because gradability and polarity do not address this aspect Of subjectivity, yet the annotators DID pay attention to this. But this only serves as noise – something that reduced the results. For applications where only evaluation is being sought, Could annotation just that and eliminate that noise – e.g., Review mining, ideological point of view, flames. But again, this Reduced the results, not increased it. Got results nonetheless.