1. Learning Subjective Adjectives From Corpora
Janyce M. Wiebe
New Mexico State University
Office of Naval Research grant N

2. Introduction Learning evaluation and opinion clues
Distributional similarity process
Refinement with lexical features
Improved results from both
Work involves learning clues of evaluation and opinion in text
Corpus-based work, meaning these clues are learned from bodies of text.
I use the results of a distributional similarity process, seeded by a small
amount of human annotation
And then refine those features with lexical semantic features which can also be
learned from corpora
Found that features based on their intersection have higher precision than features based on each alone

3. Banfield 1985, Fludernik 1993, Wiebe 1994
Subjectivity Tagging
Recognizing opinions and evaluations
(Subjective sentences) as opposed to
material objectively presented as true
(Objective sentences)
Part of a larger project investigating subjectivity, which involves aspects of language used to express opinions and evaluations.
Subjectivity tagging is distinguishing sentences used to present
opinions and evaluations from sentences used to objectively present factual material..
For pointers to work on subjectivity, you can look at these
citations in the paper.
Banfield 1985, Fludernik 1993, Wiebe 1994

4. Examples
At several different levels, it’s a fascinating tale. subjective
Bell Industries Inc. increased its quarterly to 10 cents from 7 cents a share.
objective
Example of a simple subjective sentence
Example of a simple objective sentence

5. Types ? “Enthused” “Wonderful!” “Great product” “Complained”
“Speculated”
“Maybe”
“Complained”
“You Idiot!”
“Terrible product”
This work focuses mainly on three types of subjectivity.
Negative evaluation, positive evaluation, and speculation
Clear examples of negative subjectivity:
Why the term subjectivity? Which is due to the linguist Ann Banfield.
The intuition is …

6. Subjectivity ? “Enthused” “Wonderful!” “Great product” “Complained”
“You Idiot!”
“Terrible product”
“Speculated”
“Maybe”
Need to attribute the evaluation, emotion, etc. to someone
A “subject”.
The person whose evaluation, emotion, uncertainty is being expressed.
Many different varieties within the subtypes – showing emotion
for the sake of the pictures.
If they pursue:
Can tease things apart most clearly in novels. Objective narration.
Also reporters and people who purport to objectivity.
Like any aspect of language, clear prototypical instances, and lots of borderline
Areas. Interesting to explore where people agree and don’t, which we have and are in the future doing.

7. Subjectivity Same word, different types “Great majority” objective
“Great!“ positive evaluative
“Just great.” negative evaluative
Lots of complexities which make subjectivity interesting
and non-trivial.
For one thing, context is important.
The word great, for example, can be used in an objective expressions,
as in the the great majority of seagulls can fly.
Obviously, it can be used positively.
But it can also be used negatively, as in great.

8. “It’s the best!”, he gushed.
Subjectivity
Multiple types, sources, targets
“It’s the best!”, he gushed.
As well, there might be multiple types, sources, and targets of subjectivity.
The speaker “he” is expressing positive evaluation, but simultaneously, by using “gushed” instead of something neutral like “said”, the writer could be expressing negative subjectivity.
Writer He It

9. Applications: Flame Recognition
R [ 13: Julia ] Re: BILL WARRINER!!!!
R [ 19: Suzanne ] Re: BILL WARRINER!!!!
RS < 16: Suzanne > Re: BILL WARRINER!!!!
R [ 26: Doug Bone & Jacqui D] Re: A bin full of buds
R [ 24: Karin Adamczyk ] Rose hips?
R [ 88: Colette Tremblay ] Re: Rose hips? (long)
R [ 8: Karin Adamczyk ]
R [ 29: Kay Cangemi ] Re: Rose hips?
R [ 23: Karin Adamczyk ]
R [ 30: Karin Adamczyk ]
R [ 18: BCD ] Re: red(as in wine red) roses
R [ 32: Laura Johnson-Kelly ]
RS [ 3: PattReck ]
R [ 27: Bugman ] Re: BILL WARRINER!!!!
R [ 37: Bill ]
R < 41: Celeste >
Subjectivity processing could be used in many text-processing
Flames in newsgroups and listservs are characterzied by strong negative
Evaluation directed toward another poster. The goal would be to alert the
Reader to the presence of flames.

10. Review Mining From: Hoodoo>hoodooBUGZAPPER@newnorth.net>
Newsgroups: rec.gardens
Subject: Re: Garden software
I bought a copy of Garden Encyclopedia from Sierra.
Well worth the time and money.
Filter through newsgroups and listserv discussions for reviews of products
-- information retrieval of reviews. For example, here is a positive
Review of a software product.

11. Information Extraction
Northwest Airlines settled the remaining lawsuits, a federal judge said. objective
“The cost of health care is eroding our standard of living and sapping industrial strength,” complains Maher. subjective
Here are two similar sentences, in that both are speech sentences,
But the first is objective while the second is subjective. Here, this is
being presented as factual, with a federal judge the source of information,
And it could be treated that way by an information extraction system.
But this sentence is merely presenting a complaint – we don’t want to treat
This as a fact, with the complainer a source of information.

12. Other Applications Clustering documents by ideology Text summarization
Style in machine translation and generation
There are other applications for which subjectivity tagging could be exploited.

13. Overview Identify large set of candidate clues
Existing resources are not sufficient
Not consistently marked for subjectivity
Not customized to the genre
Learn lexical clues from corpora
In this phase of the project, we are focusing on identifying
Candidate clues, words that have common subjectivity
Uses like great. Our idea is to generate a large set of candidates,
To be customized later for particular uses.
Existing lexical resources are useful, and we plan to use them, but
They aren’t subjective. Dictionaries are not consistently marked
For subjectivity. As well, they aren’t customized to the genre.
We are interested in applications in newsgroups and listservs and have done some subjectivity annotation in those genres. There is a wide variety and creativity
in subjectivity clues in those genres.
So, this project is to learn clues from corpora, or bodies of text.

14. Wiebe et al. 1999; Bruce & Wiebe 1999
Corpus and Annotation
Subjectivity tags assigned by multiple annotators to 1001 WSJ sentences
Tags representing consensus opinions obtained with EM
The data used in this work is a set of wall street journal sentences.
In previous work, multiple humans classified the sentences as either subjective or objective.
There was a round of discussion and revision of the coding manual, and
Then The humans reached good agreement. Details are in this paper.
The tags used in the current work consensus opinions of the judges. They
Were obtained using the expectation maximization algorithm as described in this paper.
We found in previous work (point to the second paper) that simply whether an adjective appears in the sentence is useful for recognizing subjective sentences.
So, in this paper, we take the performance of a simple adjective feature as a baseline, and work on identifying higher-quality adjective features for
subjectivity tagging.
Wiebe et al. 1999; Bruce & Wiebe 1999

15. Adjectives Classifications correlated with adjectives
Adjectives extracted from annotations
They promised [e+ 2 yet] more for
[e+ 3 really good] [e? 1 stuff].
"It's [e? 3 really] [e- 3 bizarre]," says Albert Lerman, creative director at the Wells agency.
More detailed manual annotations were performed for this work.
The annotators were asked to identify the expressions that made the
Subjective sentences subjective. They indicated strength and type
of subjectivity.
Go through the sentences.
Since we are focusing on adjectives,
we extracted the adjectives from the
most strongly rated expressions of one annotator.
These were used as seeds to the distributional similarity process I’ll talk about next.
If a related question:
we have plans to take advantage of these contextual influences as in word-sense disambiguation. Here we are demonstrating a process for finding large candidate classes.

16. Lin’s Distributional Similarity
have a
brown
dog R1 R3 R2 R4
Word R W
I R1 have
have R2 dog
brown R3 dog
. . .
There are a number of methods for distributional similarity that have been
used in NLP. I used Dekang Lin’s work.
The general idea is to assess
word similarity based on the distributional pattern of words.
First, the corpus
Is parsed with Lin’s dependency grammar, and these dependencies
Triples are extracted. (run your finger both ways over the arrows).
Lin’s work was good for me because he processes many different
kinds of Relations, including those involving adjectives.
Lin 1998

17. Lin’s Distributional Similarity
Word1
Word2
R W R W R W R W
R W R W
To assess the similarity between word1 and word2,
Look at the words correlated with Word1; the words correlated with
Word2. Intuitively, the more overlap there is between these sets, the more
similar the words are judged to be. The similarity metric takes into account
not only the size of the set, but also the mutual information between each word and
the members of the intersection set.
For reference:
More precisely, sum the mutual information measures of each word with members of the intersection
And divide by the sum of the mutual information measures of the word
And it’s correlated words.

18. Bizarre strange similar scary unusual fascinating
interesting curious tragic different
contradictory peculiar silly sad absurd
poignant crazy funny comic compelling
odd
Often, distributional similarity is used to aid syntactic processing and for
Semantic similarity .
But many people have pointed out that words with distributional similarity need not
Be synonyms. For example, nurse and doctor are not synonyms, but they are
often used in similar contexts.
So I was interested in the idea that the process would find words that are
Similar to their evaluative or speculative uses, even though they are not
Strictly synonymous.
So I used the adjectives extracted from the detailed annotations as seeds, an
Extracted up to 20 most similar words according to Lin’s results.
Just to look at a couple examples:
Here are the words for bizarre.
For reference: L. Lee: p(V|N); uses it for syntax, smoothing when data is sparse.

19. Bizarre strange similar scary unusual fascinating
interesting curious tragic different
contradictory peculiar silly sad absurd
poignant crazy funny comic compelling
odd
Pretty close synonyms – difference in degree.
Up to 20.

20. Bizarre strange similar scary unusual fascinating
interesting curious tragic different
contradictory peculiar silly sad absurd
poignant crazy funny comic compelling
odd
Some that are not synonyms, but are similar in that they are evaluative
(Note to me: these are not antonyms)

21. Bizarre strange similar scary unusual fascinating
interesting curious tragic different
contradictory peculiar silly sad absurd
poignant crazy funny comic compelling
odd
Some we probably want to weed out. In most contexts, only mildly evaluative and may easily appear in objective sentences.

22. Good bad better best nice poor terrific great decent lousy
dismal excellent positive exciting
fantastic marvelous strong
important dumb fair healthy
Lots of pretty close synonyms.

23. Good bad better best nice poor terrific great decent lousy
dismal excellent positive exciting
fantastic marvelous strong
important dumb fair healthy
For these we see similarly evaluative words that are antonyms
Here we see antonyms/opposites – which are evaluative in the
Opposite direction. Probably good clues of subjectivity.

24. Experiments 9 10
So, the idea is to seed the distributional similarity clustering process with adjectives extracted from subjective expressions.
To test the process multiple times, I did 10-fold cross validation.
Viewing this is a mostly supervised process – idea is to seed the clustering process with information extracted from a small amount of detailed manual annotation. So, I used 1/10 training and 9/10.

25. Experiments Distributional similarity Separate corpus Seeds Seeds +
Words 9 10
To test the process multiple times, I did 10-fold cross validation.
Viewing this is a mostly supervised process – idea is to seed the clustering process with information extracted from a small amount of detailed manual annotation. So, I used 1/10 training and 9/10.
On each fold, adjectives were extracted from the subjective elements
And the top 20 most similar words were identified, as I mentioned above.

26. Experiments Distributional similarity Separate corpus Seeds Seeds +
Words
Filtering 9 10 S
Words
p(subjective | s)
Some simple filtering was performed, based
Only on the training set. Words were removed for which the results are not better than the baseline adjective feature.
The final set was evaluated on the test set – the entire corpus minus the 1/10 of the data from which the seeds were selected.
P(sentence is subjective | one of these words appears) is on average 7.5% points higher than adjective, and 17.7% points than baseline of choosing the most frequent class.
S > Adj > Majority

27. Lexical features Polarity and Gradability Learned from corpora
Statistical processing informed by linguistic insights
Different data sets used
Then certain lexical semantic features of adjectives were considered,
Namely polarity and gradability.
Different training and testing sets were used than the corpus I used in this work.

28. Hatzivassiloglou & Wiebe 2000
Gradability
Norm Large Larger Largest
* More additional
* Very additional
Linguistic feature of adjectives – gradable or non-gradable.
A gradable adjective participates in comparative constructions
And accepts modifying expressions that act as intensifiers or diminishers.
Gradable adjectives express properties in varying degrees of strength,
Relative to a norm. Likely to be good predictors of subjectivity, given
The judgement this involves.
In contrast, the non-gradable adjective “additional” cannot be used in this way: for additional, there is no inflected comparative form, and
*more additional, and does not involve comparison relative to a norm.
Things are not all or nothing. Non-gradable adjectives can appear
Modified by grading words, and gradable adjectives can appear non-inflected and not modified by intensifiers. SO,
A statistical model was developed to classify adjectives as gradable
Or non-gradable taking into account
the number of times in a corpus an adjective
Appears in a comparative or superlative form, and the number of times
It is modified by an intensifier or diminisher like “very”.
Question they may ask: these are words that have main gradable/non-gradable usages. It is not that all instances of a word are as a gradable (or non-gradable)
Adjective. Example: “more suburban” or “very dead”.
Hatzivassiloglou & Wiebe 2000

29. Hatzivassiloglou & McKeown 1997
Polarity
- polarity
+ polarity
object
beautiful
ugly
Corrupt and brutal
* Corrupt but brutal
Semantic orientation is connected with judgements, it appeared to be a
Promising feature for subjectivity.
Exploited Linguistic observations: such as that adjectives of the same orientation
Are conjoined by conjunctions like and but not but.
Developed a statistical method for clustering adjectives into
Positive and negative polarity adjectives.
The adjectives had been identified to have orientation.
Gradability learning algorithm applied to all adjectives in the corpus.
A future experiment we plan is to run the methods in sequence – first
Gradability, then cluster those into postive and negative polarity.
The good results with the intersections of those sets – see next – make this promising.
Hatzivassiloglou & McKeown 1997

30. Experiments Separate corpus Distributional similarity Seeds Seeds +
Words
Filtering 9 10
This is the process we saw before, with the final result of filtering
called S

31. Experiments Separate corpus Distributional similarity Seeds Seeds +
Words
Filtering 9 10
For each of the lexical classes, which were learned from
a separate corpus, they are intersected with the earlier set S, and the
Precision of the resulting set is measured on the test set.
Lexical
Classification
Separate
corpus

32. Results Lex Seed Lex Pol+ + 4.6 +10.8 Pol- +18.5 +18.7
Grad
Shows the difference in precision from the baseline
Adjective feature.
Seed +7.5

33. Results Lex Seed Lex Pol+,Grad+ + 6.4 +18.0 Pol-, Grad+ +19.9 +21.4
Shows difference from the baseline adj feature for each
Seed +7.5

34. Future Work Apply process to Netnews and Listservs
Apply word-sense disambiguation techniques to potentially subjective expressions
Flame recognition and review mining
Rerunning the gradability and distributional similarity processes with Lin and Hatzivasiloglou on netnews and listserv data. Such automatic techniques probably will pay off even more in that genre, because probably less coverage of evaluative/insulting terms in standard dictionaries. People are very creative in their use of subjectivity in newsgroups and listservs.
disambiguation
Question: why not just directly use the linguistic indicators of orientation and gradability? That’s something to experiment with. Note that statistical models were developed to exploit the linguistic indicators; in the case of polarity, a clustering algorithm was used to cluster the adjectives into positive and negative polarity adjectives. Whether you want to do this as a preprocessing step as we did or part of the main classifier probably depends on the learning method used.

35. Conclusions
Learning linguistic knowledge from corpora for a pragmatic task
Linguistic information
Manual annotation
Linguistic constraints
Processes improve each other
Learning linguistic knowledge from corpora is an active area of NLP research. Much of it focuses on syntactic and semantic information; this work is a case study of learning knowledge useful for a pragamatic task, subjectivity tagging.
Statistic processing, but using linguistic information.
Used linguistic annotation and insights throughout to support/seed statistical processing.
Linguistic information informs statistical processing
Linguistic constraints used to learn lexical features
Detailed annotations seed clustering process

36. Application 1: Flame recognition
From: (PattReck)
Newsgroups: rec.gardens.roses
Subject: Re: red(as in wine red) roses
My two favorite old reds: Cramoisi Superieure, especially great
climbing, and Francis Dubreuil. Also Prospero does well in
southern California - aren't you on the west coast? -- Candace

37. Flames (continued) From: Suzanne <Suzanne_member@newsguy.com>
Newsgroups: rec.gardens.roses
Subject: Re: BILL WARRINER!!!!
>>Wow. You guys are really working poor Suzanne over.
>po thang. I thank she been workin over her bottle of Kahlua.
***Up &^%$!!! I've been working at a *job* - no Kahlua! You are a
snow-snorting dust-bowl dweller, the dustiest of the dusties. Bill
Bradley has the support of the "environmentalists" ha ha ha!

38. Likely likely possible willing probable
receptive unlikely able logical
rumored potential counterproductive
moot significant hesitant worthy
unwilling probably desirable
weak forthcoming imminent
Noise in our data, because these tended to get filtered out,
Because gradability and polarity do not address this aspect
Of subjectivity, yet the annotators DID pay attention to this.
But this only serves as noise – something that reduced the results.
For applications where only evaluation is being sought,
Could annotation just that and eliminate that noise – e.g.,
Review mining, ideological point of view, flames. But again, this
Reduced the results, not increased it. Got results nonetheless.