Presentation is loading. Please wait.

Presentation is loading. Please wait.

Annotating Opinions in the World Press Theresa Wilson and Janyce Wiebe University of Pittsburgh Intelligent Systems Program and Department of Computer.

Published byVivien Jones Modified over 9 years ago

Similar presentations

Presentation on theme: "Annotating Opinions in the World Press Theresa Wilson and Janyce Wiebe University of Pittsburgh Intelligent Systems Program and Department of Computer."— Presentation transcript:

1 Annotating Opinions in the World Press Theresa Wilson and Janyce Wiebe University of Pittsburgh Intelligent Systems Program and Department of Computer Science

2 July 5, 2003SIGdial 20032 Introduction Private State: mental or emotional state, which cannot be directly observed or verified (Quirk et al., 1985) Emotions Speculation Beliefs Opinions Evaluations

3 July 5, 2003SIGdial 20033 Private States in Text Subjectivity: aspects of language used to express private states, including opinions, emotions, evaluations, speculations, sentiments (Banfield 1982) “Complained” “You Idiot!” “Terrible product” “Speculated” “Maybe” “I wonder” “Enthused” “Wonderful!” “Great idea” (See Fodor 1979, Cohn 1978, Chafe 1986, Fludernik 1993, Wiebe 1994, Stein & Wright 1995 for additional references.)

4 July 5, 2003SIGdial 20034 Why do we care? News articles mix of factual/subjective language News articles mix of factual/subjective language Help existing NLP applications: IE, IR, summarization systems, etc. Help existing NLP applications: IE, IR, summarization systems, etc. Support new tasks: multi-perspective question answering Support new tasks: multi-perspective question answering

5 July 5, 2003SIGdial 20035 Direct mentions of private states Direct mentions of private states The United States fears a spill-over from the anti-terrorist campaign. Expressive subjective elements Expressive subjective elements The part of the US human rights report about China is full of absurdities and fabrications. Two Main Ways of Expressing Private States

6 July 5, 2003SIGdial 20036 Private States and Speech Events Speech Event  speaking or writing Speech Event  speaking or writing “We foresaw electoral fraud but not daylight robbery,” Tsvangirai said. Medical Department head Dr. Hameed Saeed said the patient’s blood has been sent to the Institute for Virology in Johannesburg for analysis.

7 July 5, 2003SIGdial 20037 Nested Sources “The US fears a spill-over’’, said Xirao-Nima, a professor of foreign affairs at the Central University for Nationalities. (writer, Xirao-Nima, US)(writer, Xirao-Nima)(writer) “The report is full of absurdities,’’ he continued. (writer, Xirao-Nima) (writer)

8 July 5, 2003SIGdial 20038 Annotation Scheme Implemented in GATE (http://gate.ac.uk) Implemented in GATE (http://gate.ac.uk)http://gate.ac.uk Shorthand for direct private state or speech event expression

9 July 5, 2003SIGdial 20039 Attributes for Annotation Types On (Direct Private State or Speech Event) nested-source nested-source onlyfactive onlyfactive overall-strength overall-strength on-strength on-strength attitude-type attitude-type attitude-toward attitude-toward is-implicit is-implicit minor minor Expressive Subjective Element nested-source strength attitude-type

10 July 5, 2003SIGdial 200310 Attributes for Annotation Types On (Direct Private State or Speech Event)  nested-source onlyfactive onlyfactive overall-strength overall-strength on-strength on-strength attitude-type attitude-type attitude-toward attitude-toward is-implicit is-implicit minor minor Expressive Subjective Element  nested-source strength attitude-type

11 July 5, 2003SIGdial 200311 Attributes for Annotation Types On (Direct Private State or Speech Event)  nested-source onlyfactive onlyfactive overall-strength overall-strength on-strength on-strength  attitude-type  attitude-toward  is-implicit  minor Expressive Subjective Element  nested-source strength  attitude-type

12 July 5, 2003SIGdial 200312 Attributes for Annotation Types On (Direct Private State or Speech Event)  nested-source  onlyfactive  overall-strength  on-strength  attitude-type  attitude-toward  is-implicit  minor Expressive Subjective Element  nested-source  strength  attitude-type

13 July 5, 2003SIGdial 200313 onlyfactive Attribute “The US fears a spill-over’’, said Xirao-Nima, a professor of foreign affairs at the Central University for Nationalities. (writer) onlyfactive=yes (writer, Xirao-Nima) onlyfactive=yes (writer, Xirao-Nima, US) onlyfactive=no

14 July 5, 2003SIGdial 200314 Important! onlyfactive=no may still contain facts “John criticized Mary for smoking.” (writer, John) onlyfactive=no

15 July 5, 2003SIGdial 200315 The Foreign Ministry said... on-strength=neutral The Foreign Ministry objected... The Foreign Ministry vehemently denied... on-strength=medium Strength of Private State on-strength=high... inaccurate section on Chechnya. strength=low... odious section on Chechnya. strength=high

16 July 5, 2003SIGdial 200316 Overall Strength of Private State overall-strength=high The Foreign Ministry objected in particular to the “odious” section on Chechnya. on-strength=mediumstrength=high “We foresaw electoral fraud but not daylight robbery,” Tsvangirai said. strength=medium on-strength=neutral overall-strength=medium

17 July 5, 2003SIGdial 200317 Data English-language versions of articles from world press (FBIS + few other sources) English-language versions of articles from world press (FBIS + few other sources) 8500 sentences have been annotated 8500 sentences have been annotated Agreement Study Agreement Study 3 annotators (A, M, S) 3 annotators (A, M, S) 210 sentences (13 documents) 210 sentences (13 documents) 1/3 news, 1/3 news on “hot-topic”, 1/3 editorials 1/3 news, 1/3 news on “hot-topic”, 1/3 editorials

18 July 5, 2003SIGdial 200318 Types of Agreement Agreement for spans Agreement for spans Ons Ons Expressive Subjective Elements Expressive Subjective Elements Agreement for ON  onlyfactive attribute Agreement for ON  onlyfactive attribute Agreement for sentences Agreement for sentences Derive sentence-level annotations from lower- level annotations Derive sentence-level annotations from lower- level annotations

19 July 5, 2003SIGdial 200319 Agreement Issues: Spans Overlapping spans ok  it’s a match Overlapping spans ok  it’s a match A: His comments were immediately contested... M: His comments were immediately contested...

20 July 5, 2003SIGdial 200320 Agreement Issues: Spans A: Russian military policy has been to keep a distance from... M: Russian military policy has been to keep a distance from... (writer, Russian Miliatary) Different set of annotations for each annotator  Kappa inappropriate Different set of annotations for each annotator  Kappa inappropriate

21 July 5, 2003SIGdial 200321 Annotator B Annotator A Agreement Evaluation: spans Agreement measured by percentage of A’s constituents matching B’s, and vice versa Agreement measured by percentage of A’s constituents matching B’s, and vice versa agr (a||b) = |A matching B| |A||A|

22 July 5, 2003SIGdial 200322 Span Agreement: Ons abagr(a||b)agr(b||a)average AM0.750.91 AS0.800.85 MS0.860.75 0.82

23 July 5, 2003SIGdial 200323 Measuring Agreement: ON  onlyfactive A:  Reformers told Putin all along that he should take advantage of "the rare opportunity for (Russia) to become integrated with the civilized world."  But main stream Russian military policy has been to keep a distance from America. M:  Reformers told Putin all along that he should take advantage of "the rare opportunity for (Russia) to become integrated with the civilized world.“  But main stream Russian military policy has been to keep a distance from America.   writer on

24 July 5, 2003SIGdial 200324 Measuring Agreement: ON  onlyfactive A:  Reformers told Putin all along that he should take advantage of "the rare opportunity for (Russia) to become integrated with the civilized world."  But main stream Russian military policy has been to keep a distance from America. M:  Reformers told Putin all along that he should take advantage of "the rare opportunity for (Russia) to become integrated with the civilized world.“  But main stream Russian military policy has been to keep a distance from America.   writer on

25 July 5, 2003SIGdial 200325 Measuring Agreement: ON  onlyfactive A:  Reformers told Putin all along that he should take advantage of "the rare opportunity for (Russia) to become integrated with the civilized world."  But main stream Russian military policy has been to keep a distance from America. M:  Reformers told Putin all along that he should take advantage of "the rare opportunity for (Russia) to become integrated with the civilized world.“  But main stream Russian military policy has been to keep a distance from America.   writer on onlyfactive=yesonlyfactive=no onlyfactive=yes onlyfactive=no onlyfactive=yes

26 July 5, 2003SIGdial 200326 Agreement results: onlyfactive Borderline onlyfactive: Borderline onlyfactive: 1. At least one annotator marked expression onlyfactive=no 2. Neither annotator marked expression overall-strength > low Ave Kappa

27 July 5, 2003SIGdial 200327 Deriving Sentence-level Annotations Sue said, “The election was Tuesday.” (writer) onlyfactive=yes (writer, Sue) onlyfactive=yes Objective Sue said, “The election was fair.” (writer) onlyfactive=yes (writer, Sue) onlyfactive=no Subjective

28 July 5, 2003SIGdial 200328 Agreement results: sentence-level subjective annotations Borderline subjective Borderline subjective sentence: sentence: 1. At least one annotator marked at least one expression onlyfactive=no 2. Neither annotator marked an overallstrength > low Ave Kappa

29 July 5, 2003SIGdial 200329 Conclusions Detailed scheme for annotation of opinions and other private states Detailed scheme for annotation of opinions and other private states  Good agreement between annotators  Strong agreement about clear cases of subjectivity Lower-level annotations  higher-level subjective sentence annotations Lower-level annotations  higher-level subjective sentence annotations Extendable Extendable

30 July 5, 2003SIGdial 200330 Using GATE to perform private state annotations GATE GATE http://gate.ac.uk General annotation instructions and GATE instructions General annotation instructions and GATE instructions http://www.cs.pitt.edu/~wiebe/pubs/ardasummer02

31 July 5, 2003SIGdial 200331 Thank you This work was performed in support of the Northeast Regional Research Center (NRRC) which is sponsored by the Advanced Research and Development Activity in Information Technology (ARDA), a U.S. Government entity which sponsors and promotes research of import to the Intelligence Community which includes but is not limited to the CIA, DIA, NSA, NIMA and NRO.

32 July 5, 2003SIGdial 200332 Contingency Table: onlyfactive judgements Tagger M YesNo TaggerAYes n yy =181 n yn =25 No n ny =12 n nn =252 Agreement for onlyfactive judgments Tagger M YesNo TaggerAYes n yy =181 n yn =8 No n ny =11 n nn =224 Agreement for onlyfactive judgments, borderline cases removed

33 July 5, 2003SIGdial 200333 Onlyfactive Agreement All Expressions Borderline Removed KappaagreeKappaagree % removed A & M 0.840.910.940.9610 A & S 0.840.920.900.958 M & S 0.740.870.840.9212

34 July 5, 2003SIGdial 200334 Subjective Sentence Agreement All Expressions Borderline Removed KappaagreeKappaagree % removed A & M 0.750.890.870.9511 A & S 0.840.940.920.978 M & S 0.720.880.830.9313

35 July 5, 2003SIGdial 200335 Why we exclude minor on annotations This is the first time that the Cuban government has expressed its position over the U.S. holding of the Taliban and al-Qaeda prisoners in its territory. (writer, Cuban gov) onlyfactive=no Objective Sentence!

Download ppt "Annotating Opinions in the World Press Theresa Wilson and Janyce Wiebe University of Pittsburgh Intelligent Systems Program and Department of Computer."

Similar presentations

of English and Journalism Cuyahoga Community College

of English and Journalism Cuyahoga Community College
(look at your chart). The form of nonfiction refers to its structure The authors is the structural genre the author uses There are 5 main forms of Nonfiction.

(look at your chart). The form of nonfiction refers to its structure The authors is the structural genre the author uses There are 5 main forms of Nonfiction.
An In-Depth Look at the Synthesis Essay Question Preparing for the AP Language and Composition Exam.

An In-Depth Look at the Synthesis Essay Question Preparing for the AP Language and Composition Exam.
By: Mrs. Wilson.  Main Idea and Details  Problem and Solution  Logical Order  Summarizing  Author’s Purpose.

By: Mrs. Wilson.  Main Idea and Details  Problem and Solution  Logical Order  Summarizing  Author’s Purpose.
Evaluation Questions The new style of question from 2012 onwards...

Evaluation Questions The new style of question from 2012 onwards...
Manual Subjectivity Analysis. EUROLAN July 30, 20072 Preliminaries What do we mean by subjectivity? The linguistic expression of somebody’s emotions,

Manual Subjectivity Analysis. EUROLAN July 30, Preliminaries What do we mean by subjectivity? The linguistic expression of somebody’s emotions,
Playing the Telephone Game: Determining the Hierarchical Structure of Perspective and Speech Expressions Eric Breck and Claire Cardie Department of Computer.

Playing the Telephone Game: Determining the Hierarchical Structure of Perspective and Speech Expressions Eric Breck and Claire Cardie Department of Computer.
A Brief Overview. Contents Introduction to NLP Sentiment Analysis Subjectivity versus Objectivity Determining Polarity Statistical & Linguistic Approaches.

A Brief Overview. Contents Introduction to NLP Sentiment Analysis Subjectivity versus Objectivity Determining Polarity Statistical & Linguistic Approaches.
Annotating Topics of Opinions Veselin Stoyanov Claire Cardie.

Annotating Topics of Opinions Veselin Stoyanov Claire Cardie.
Multi-Perspective Question Answering Using the OpQA Corpus Veselin Stoyanov Claire Cardie Janyce Wiebe Cornell University University of Pittsburgh.

Multi-Perspective Question Answering Using the OpQA Corpus Veselin Stoyanov Claire Cardie Janyce Wiebe Cornell University University of Pittsburgh.
Annotating Expressions of Opinions and Emotions in Language Wiebe, Wilson, Cardie.

Annotating Expressions of Opinions and Emotions in Language Wiebe, Wilson, Cardie.
“Quotations” and Attribution Chapter 9. “Why Use Quotes?” he asked. Because quotations add color and interest to news stories by allowing readers to “hear”

“Quotations” and Attribution Chapter 9. “Why Use Quotes?” he asked. Because quotations add color and interest to news stories by allowing readers to “hear”
Subjectivity Annotation Update Josef Ruppenhofer Jan Wiebe.

Subjectivity Annotation Update Josef Ruppenhofer Jan Wiebe.
1 Attributions and Private States Jan Wiebe (U. Pittsburgh) Theresa Wilson (U. Pittsburgh) Claire Cardie (Cornell U.)

1 Attributions and Private States Jan Wiebe (U. Pittsburgh) Theresa Wilson (U. Pittsburgh) Claire Cardie (Cornell U.)
Combining Low-Level and Summary Representations of Opinions for Multi- Perspective Question Answering Claire Cardie, Janyce Wiebe, Theresa Wilson, Diane.

Combining Low-Level and Summary Representations of Opinions for Multi- Perspective Question Answering Claire Cardie, Janyce Wiebe, Theresa Wilson, Diane.
Learning Subjective Adjectives from Corpora Janyce M. Wiebe Presenter: Gabriel Nicolae.

Learning Subjective Adjectives from Corpora Janyce M. Wiebe Presenter: Gabriel Nicolae.
Introduction to Nonfiction

Introduction to Nonfiction
Magazine Editor By: Eunice Kim.

Magazine Editor By: Eunice Kim.
CLUE Framework IETF 84 July 30 – Aug 3, 2012 Mark Duckworth Allyn Romanow Brian Baldino Andy Pepperell.

CLUE Framework IETF 84 July 30 – Aug 3, 2012 Mark Duckworth Allyn Romanow Brian Baldino Andy Pepperell.
Writing Across the Curriculum Collins’ Writing. To develop successful, life-long writers, students must have: Opportunities to: write in many environments.

Writing Across the Curriculum Collins’ Writing. To develop successful, life-long writers, students must have: Opportunities to: write in many environments.

Similar presentations

Ads by Google