Annotating Topics of Opinions Veselin Stoyanov Claire Cardie

May 29, 2008 LREC 2008, Marakech, Morocco Talk Overview Fine-grained sentiment analysis  Definitions  Examples Opinion topic annotation  Definitions  Issues  Approach and Corpus  IA agreement

May 29, 2008 LREC 2008, Marakech, Morocco Background Sentiment Analysis: Extraction and representation of attitudes, evaluations, opinions, and sentiment in text. Fine-grained Sentiment Analysis: At the level of individual expressions of opinions.

May 29, 2008 LREC 2008, Marakech, Morocco The Australian press has launched a bitter attack on Italy after seeing their beloved Socceroos eliminated on a controversial late penalty. Italian coach Lippi has been blasted for his favorable comments toward the penalty. Lippi is preparing his side for the upcoming clash with Ukraine. He hailed 10-man Italy's determination to beat Australia and reiterated that the penalty was rightly given. Fine-grained vs. Coarse-grained Sentiment Analysis  Coarse-grained Sentiment classification Useful in the product review domain  Fine-grained Individual expressions of opinions Multiple opinions per document (even sentence) Review 1 Review 2 Positive Negative [ S The Australian press] has launched a bitter attack on [ T Italy] after seeing their beloved [ T Socceroos] eliminated on a controversial late [ T penalty]. [ S+T Italian coach Lippi] has also been blasted for his favorable comments toward [ T the penalty]. Lippi is preparing his side for the upcoming clash with Ukraine. [ S He] hailed 10-man [ T Italy]'s determination to beat Australia and reiterated that the [ T penalty] was rightly given.

May 29, 2008 LREC 2008, Marakech, Morocco Fine-grained opinions: Example The Australian press has launched a bitter attack on Italy.

May 29, 2008 LREC 2008, Marakech, Morocco Fine-grained opinions: Example  Opinion trigger (opinion words)  Source (opinion holder)  Polarity – positive/negative  Strength  Topic (target) The Australian press has launched a bitter attack on Italy. Definitions differ, but five main components:

May 29, 2008 LREC 2008, Marakech, Morocco Fine-grained opinions: Example  Opinion trigger (opinion words)  Source (opinion holder)  Polarity – positive/negative  Strength  Topic (target) The Australian press has launched a bitter attack on Italy. Definitions differ, but five main components: launched a bitter attack

May 29, 2008 LREC 2008, Marakech, Morocco Fine-grained opinions: Example  Opinion trigger (opinion words)  Source (opinion holder)  Polarity – positive/negative  Strength  Topic (target) [ S The Australian press] has launched a bitter attack on Italy. Definitions differ, but five main components: launched a bitter attack The Australian press

May 29, 2008 LREC 2008, Marakech, Morocco Fine-grained opinions: Example  Opinion trigger (opinion words)  Source (opinion holder)  Polarity – positive/negative  Strength  Topic (target) [ S The Australian press] has launched a bitter attack on Italy. Definitions differ, but five main components: launched a bitter attack The Australian press negative

May 29, 2008 LREC 2008, Marakech, Morocco Fine-grained opinions: Example  Opinion trigger (opinion words)  Source (opinion holder)  Polarity – positive/negative  Strength  Topic (target) [ S The Australian press] has launched a bitter attack on Italy. Definitions differ, but five main components: launched a bitter attack The Australian press negative high

May 29, 2008 LREC 2008, Marakech, Morocco Fine-grained opinions: Example  Opinion trigger (opinion words)  Source (opinion holder)  Polarity – positive/negative  Strength  Topic (target) [ S The Australian press] has launched a bitter attack on [ T Italy] Definitions differ, but five main components: launched a bitter attack The Australian press negative high Italy

May 29, 2008 LREC 2008, Marakech, Morocco Fine-grained opinions Five components  Source (opinion holder) e.g. [Bethard et al., 2004] [Choi et al., 2005] [Kim and Hovy, 2006]  Opinion trigger (opinion words) e.g. [Yu and Hatzivassiloglou, 2003] [Riloff and Wiebe, 2003]  Polarity – positive/negative As above  Strength e.g. [Wilson et al. 2004]  Topic (target) ????

May 29, 2008 LREC 2008, Marakech, Morocco Annotating Topics of Fine-grained Opinions  Definitions  Issues  Approach and Corpus  IA agreement

May 29, 2008 LREC 2008, Marakech, Morocco Examples (1)[ OH John] likes Marseille for its weather and cultural diversity. (2)[ OH Al] thinks that the government should tax gas more in order to curb CO2 emissions.

May 29, 2008 LREC 2008, Marakech, Morocco Definitions (1)[ OH John] likes Marseille for its weather and cultural diversity.

May 29, 2008 LREC 2008, Marakech, Morocco Definitions (1)[ OH John] likes Marseille for its weather and cultural diversity. Topic: city of Marseille Topic - the real-world object, event or abstract entity that is the subject of the opinion as intended by the opinion holder

May 29, 2008 LREC 2008, Marakech, Morocco Definitions (1)[ OH John] likes [ TOPIC SPAN Marseille] for its weather and cultural diversity. Topic: city of Marseille Topic - the real-world object, event or abstract entity that is the subject of the opinion as intended by the opinion holder Topic span - the closest, minimal span of text that mentions the topic

May 29, 2008 LREC 2008, Marakech, Morocco Definitions (1)[ OH John] likes [ TARGET+TOPIC SPAN Marseille] for its weather and cultural diversity. Topic: city of Marseille Topic - the real-world object, event or abstract entity that is the subject of the opinion as intended by the opinion holder Topic span - the closest, minimal span of text that mentions the topic Target span - the span of text that covers the syntactic surface form comprising the contents of the opinion

May 29, 2008 LREC 2008, Marakech, Morocco Definitions (2)[ OH Al] thinks that the government should tax gas more in order to curb CO2 emissions.

May 29, 2008 LREC 2008, Marakech, Morocco Definitions (2)[ OH Al] thinks that [ TARGET SPAN the government should tax gas more in order to curb CO2 emissions].

May 29, 2008 LREC 2008, Marakech, Morocco Definitions (2)[ OH Al] thinks that [ TARGET SPAN [ TOPIC SPAN? the government] should [ TOPIC SPAN? tax gas] more in order to [ TOPIC SPAN? curb [ TOPIC SPAN? CO2 emissions]]].

May 29, 2008 LREC 2008, Marakech, Morocco Definitions (2)[ OH Al] thinks that [ TARGET SPAN the government should tax gas more in order to curb CO2 emissions]. Context: (3) Although he doesn’t like government imposed taxes, he thinks that a fuel tax is the only effective solution.

May 29, 2008 LREC 2008, Marakech, Morocco Definitions (2)[ OH Al] thinks that [ TARGET SPAN the government should [ TOPIC SPAN tax gas] more in order to curb CO2 emissions]. Context: (3) Although he doesn’t like government imposed taxes, he thinks that a fuel tax is the only effective solution.

May 29, 2008 LREC 2008, Marakech, Morocco Related Work Product reviews  E.g. Kobayashi et al. (2004), Yi et al. (2003), Popescu and Etzioni (2005), Hu and Liu (2004  Limit “topics” to mentions of product names, components, and their attributes  Lexicon look-up  Focused on methods for lexicon acquisition MPQA corpus (Wiebe, Wilson, Cardie, 2004)  Fine-grained opinions  Topic annotation deemed too difficult  Target span annotation is underway Kim & Hovy (2006)  Target span extraction using semantic frames  Limited evaluation

May 29, 2008 LREC 2008, Marakech, Morocco Issues in Opinion Topic Identification Multiple potential topics mentioned within a single target span (2)[ OH Al] thinks that [ TARGET SPAN [ TOPIC SPAN? the government] should [ TOPIC SPAN? tax gas] more in order to [ TOPIC SPAN? curb [ TOPIC SPAN? CO2 emissions]]]. Requires context Topic of an opinion is the entity that comprises the main information goal of the opinion based on the discourse context.

May 29, 2008 LREC 2008, Marakech, Morocco Issues in Opinion Topic Identification Opinion topics are not always explicitly mentioned (4) [ OH John] believes the violation of Palestinian human rights is one of the main factors. Topic: ISRAELI-PALESTINIAN CONFLICT (5) [ OH I] disagree entirely!

May 29, 2008 LREC 2008, Marakech, Morocco A Coreference Approach Hypothesize that the notion of topic coreference will facilitate identification of opinion topics Easier than specifying the topic of each opinion in isolation Two opinions are topic-coreferent if they share the same opinion topic.

May 29, 2008 LREC 2008, Marakech, Morocco Opinion Topic Corpus Build on the MPQA corpus: 535 Documents manually annotated for fine- grained opinions No opinion topic annotation Our goal: Add the opinion topic information on top of the existing opinion annotations Created and used a GUI (

May 29, 2008 LREC 2008, Marakech, Morocco Annotation Process List of opinions to be processed Set of current clusters Document text

May 29, 2008 LREC 2008, Marakech, Morocco Annotation Process

May 29, 2008 LREC 2008, Marakech, Morocco Annotation Process fuel tax

May 29, 2008 LREC 2008, Marakech, Morocco Interannotator Agreement Annotator 1  150 of the 535 MPQA documents Annotator 2  20 of these 150 IAG measures from noun phrase coreference resolution B3B3  CEAF all opinions sentiment-bearing opinions strong opinions

May 29, 2008 LREC 2008, Marakech, Morocco Interannotator Agreement Annotator 1  150 of the 535 MPQA documents Annotator 2  20 of these 150 IAG measures from noun phrase coreference resolution B3B3  CEAF all opinions sentiment-bearing opinions strong opinions

May 29, 2008 LREC 2008, Marakech, Morocco Baselines all-in-one  assigns all opinions to the same cluster 1 opinion per cluster  assigns each opinion to its own cluster same paragraph  opinions in the same paragraph are assigned to the same cluster

May 29, 2008 LREC 2008, Marakech, Morocco Results Baselines vs. Interannotator agreement B3B3  CEAF all-in-one opinion per cluster same paragraph all opinions sentiment-bearing strong opinions

May 29, 2008 LREC 2008, Marakech, Morocco Thank you Questions? Annotation instructions + more information available at:

May 29, 2008 LREC 2008, Marakech, Morocco Example The Australian press has launched a bitter attack on Italy after seeing their beloved Socceroos eliminated on a controversial late penalty. Italian coach Lippi has been blasted for his favorable comments toward the penalty. Lippi is preparing his side for the upcoming clash with Ukraine. He hailed 10-man Italy's determination to beat Australia and reiterated that the penalty was rightly given.

May 29, 2008 LREC 2008, Marakech, Morocco Example – fine-grained opinions [ S The Australian press] has launched a bitter attack on [ T Italy] after seeing [ S their] beloved [ T Socceroos] eliminated on a controversial late [ T penalty]. [ S+T Italian coach Lippi] has also been blasted for his favorable comments toward [ T the penalty]. Lippi is preparing his side for the upcoming clash with Ukraine. [ S He] hailed 10-man [ T Italy]'s determination to beat Australia and reiterated that [ T the penalty] was rightly given.

May 29, 2008 LREC 2008, Marakech, Morocco Motivation Sentiment analysis: Useful as stand-alone application  Product reviews  Tracking opinions in the press  Flame detection, etc. Opinion information can benefit many NLP applications  Multi-Perspective Question Answering [Stoyanov, Cardie, Litman and Wiebe. AAAI WS 2004] and [Stoyanov, Cardie and Wiebe. HLT-EMNLP 2005]  Opinion-Oriented Information Retrieval  Clustering, etc.

May 29, 2008 LREC 2008, Marakech, Morocco Annotation Process

May 29, 2008 LREC 2008, Marakech, Morocco Annotation Process

May 29, 2008 LREC 2008, Marakech, Morocco `