Annotating Opinions in the World Press Theresa Wilson and Janyce Wiebe University of Pittsburgh Intelligent Systems Program and Department of Computer Science

July 5, 2003SIGdial Introduction Private State: mental or emotional state, which cannot be directly observed or verified (Quirk et al., 1985) Emotions Speculation Beliefs Opinions Evaluations

July 5, 2003SIGdial Private States in Text Subjectivity: aspects of language used to express private states, including opinions, emotions, evaluations, speculations, sentiments (Banfield 1982) “Complained” “You Idiot!” “Terrible product” “Speculated” “Maybe” “I wonder” “Enthused” “Wonderful!” “Great idea” (See Fodor 1979, Cohn 1978, Chafe 1986, Fludernik 1993, Wiebe 1994, Stein & Wright 1995 for additional references.)

July 5, 2003SIGdial Why do we care? News articles mix of factual/subjective language News articles mix of factual/subjective language Help existing NLP applications: IE, IR, summarization systems, etc. Help existing NLP applications: IE, IR, summarization systems, etc. Support new tasks: multi-perspective question answering Support new tasks: multi-perspective question answering

July 5, 2003SIGdial Direct mentions of private states Direct mentions of private states The United States fears a spill-over from the anti-terrorist campaign. Expressive subjective elements Expressive subjective elements The part of the US human rights report about China is full of absurdities and fabrications. Two Main Ways of Expressing Private States

July 5, 2003SIGdial Private States and Speech Events Speech Event  speaking or writing Speech Event  speaking or writing “We foresaw electoral fraud but not daylight robbery,” Tsvangirai said. Medical Department head Dr. Hameed Saeed said the patient’s blood has been sent to the Institute for Virology in Johannesburg for analysis.

July 5, 2003SIGdial Nested Sources “The US fears a spill-over’’, said Xirao-Nima, a professor of foreign affairs at the Central University for Nationalities. (writer, Xirao-Nima, US)(writer, Xirao-Nima)(writer) “The report is full of absurdities,’’ he continued. (writer, Xirao-Nima) (writer)

July 5, 2003SIGdial Annotation Scheme Implemented in GATE ( Implemented in GATE ( Shorthand for direct private state or speech event expression

July 5, 2003SIGdial Attributes for Annotation Types On (Direct Private State or Speech Event) nested-source nested-source onlyfactive onlyfactive overall-strength overall-strength on-strength on-strength attitude-type attitude-type attitude-toward attitude-toward is-implicit is-implicit minor minor Expressive Subjective Element nested-source strength attitude-type

July 5, 2003SIGdial Attributes for Annotation Types On (Direct Private State or Speech Event)  nested-source onlyfactive onlyfactive overall-strength overall-strength on-strength on-strength attitude-type attitude-type attitude-toward attitude-toward is-implicit is-implicit minor minor Expressive Subjective Element  nested-source strength attitude-type

July 5, 2003SIGdial Attributes for Annotation Types On (Direct Private State or Speech Event)  nested-source onlyfactive onlyfactive overall-strength overall-strength on-strength on-strength  attitude-type  attitude-toward  is-implicit  minor Expressive Subjective Element  nested-source strength  attitude-type

July 5, 2003SIGdial Attributes for Annotation Types On (Direct Private State or Speech Event)  nested-source  onlyfactive  overall-strength  on-strength  attitude-type  attitude-toward  is-implicit  minor Expressive Subjective Element  nested-source  strength  attitude-type

July 5, 2003SIGdial onlyfactive Attribute “The US fears a spill-over’’, said Xirao-Nima, a professor of foreign affairs at the Central University for Nationalities. (writer) onlyfactive=yes (writer, Xirao-Nima) onlyfactive=yes (writer, Xirao-Nima, US) onlyfactive=no

July 5, 2003SIGdial Important! onlyfactive=no may still contain facts “John criticized Mary for smoking.” (writer, John) onlyfactive=no

July 5, 2003SIGdial The Foreign Ministry said... on-strength=neutral The Foreign Ministry objected... The Foreign Ministry vehemently denied... on-strength=medium Strength of Private State on-strength=high... inaccurate section on Chechnya. strength=low... odious section on Chechnya. strength=high

July 5, 2003SIGdial Overall Strength of Private State overall-strength=high The Foreign Ministry objected in particular to the “odious” section on Chechnya. on-strength=mediumstrength=high “We foresaw electoral fraud but not daylight robbery,” Tsvangirai said. strength=medium on-strength=neutral overall-strength=medium

July 5, 2003SIGdial Data English-language versions of articles from world press (FBIS + few other sources) English-language versions of articles from world press (FBIS + few other sources) 8500 sentences have been annotated 8500 sentences have been annotated Agreement Study Agreement Study 3 annotators (A, M, S) 3 annotators (A, M, S) 210 sentences (13 documents) 210 sentences (13 documents) 1/3 news, 1/3 news on “hot-topic”, 1/3 editorials 1/3 news, 1/3 news on “hot-topic”, 1/3 editorials

July 5, 2003SIGdial Types of Agreement Agreement for spans Agreement for spans Ons Ons Expressive Subjective Elements Expressive Subjective Elements Agreement for ON  onlyfactive attribute Agreement for ON  onlyfactive attribute Agreement for sentences Agreement for sentences Derive sentence-level annotations from lower- level annotations Derive sentence-level annotations from lower- level annotations

July 5, 2003SIGdial Agreement Issues: Spans Overlapping spans ok  it’s a match Overlapping spans ok  it’s a match A: His comments were immediately contested... M: His comments were immediately contested...

July 5, 2003SIGdial Agreement Issues: Spans A: Russian military policy has been to keep a distance from... M: Russian military policy has been to keep a distance from... (writer, Russian Miliatary) Different set of annotations for each annotator  Kappa inappropriate Different set of annotations for each annotator  Kappa inappropriate

July 5, 2003SIGdial Annotator B Annotator A Agreement Evaluation: spans Agreement measured by percentage of A’s constituents matching B’s, and vice versa Agreement measured by percentage of A’s constituents matching B’s, and vice versa agr (a||b) = |A matching B| |A||A|

July 5, 2003SIGdial Span Agreement: Ons abagr(a||b)agr(b||a)average AM AS MS

July 5, 2003SIGdial Measuring Agreement: ON  onlyfactive A:  Reformers told Putin all along that he should take advantage of "the rare opportunity for (Russia) to become integrated with the civilized world."  But main stream Russian military policy has been to keep a distance from America. M:  Reformers told Putin all along that he should take advantage of "the rare opportunity for (Russia) to become integrated with the civilized world.“  But main stream Russian military policy has been to keep a distance from America.   writer on

July 5, 2003SIGdial Measuring Agreement: ON  onlyfactive A:  Reformers told Putin all along that he should take advantage of "the rare opportunity for (Russia) to become integrated with the civilized world."  But main stream Russian military policy has been to keep a distance from America. M:  Reformers told Putin all along that he should take advantage of "the rare opportunity for (Russia) to become integrated with the civilized world.“  But main stream Russian military policy has been to keep a distance from America.   writer on

July 5, 2003SIGdial Measuring Agreement: ON  onlyfactive A:  Reformers told Putin all along that he should take advantage of "the rare opportunity for (Russia) to become integrated with the civilized world."  But main stream Russian military policy has been to keep a distance from America. M:  Reformers told Putin all along that he should take advantage of "the rare opportunity for (Russia) to become integrated with the civilized world.“  But main stream Russian military policy has been to keep a distance from America.   writer on onlyfactive=yesonlyfactive=no onlyfactive=yes onlyfactive=no onlyfactive=yes

July 5, 2003SIGdial Agreement results: onlyfactive Borderline onlyfactive: Borderline onlyfactive: 1. At least one annotator marked expression onlyfactive=no 2. Neither annotator marked expression overall-strength > low Ave Kappa

July 5, 2003SIGdial Deriving Sentence-level Annotations Sue said, “The election was Tuesday.” (writer) onlyfactive=yes (writer, Sue) onlyfactive=yes Objective Sue said, “The election was fair.” (writer) onlyfactive=yes (writer, Sue) onlyfactive=no Subjective

July 5, 2003SIGdial Agreement results: sentence-level subjective annotations Borderline subjective Borderline subjective sentence: sentence: 1. At least one annotator marked at least one expression onlyfactive=no 2. Neither annotator marked an overallstrength > low Ave Kappa

July 5, 2003SIGdial Conclusions Detailed scheme for annotation of opinions and other private states Detailed scheme for annotation of opinions and other private states  Good agreement between annotators  Strong agreement about clear cases of subjectivity Lower-level annotations  higher-level subjective sentence annotations Lower-level annotations  higher-level subjective sentence annotations Extendable Extendable

July 5, 2003SIGdial Using GATE to perform private state annotations GATE GATE General annotation instructions and GATE instructions General annotation instructions and GATE instructions

July 5, 2003SIGdial Thank you This work was performed in support of the Northeast Regional Research Center (NRRC) which is sponsored by the Advanced Research and Development Activity in Information Technology (ARDA), a U.S. Government entity which sponsors and promotes research of import to the Intelligence Community which includes but is not limited to the CIA, DIA, NSA, NIMA and NRO.

July 5, 2003SIGdial Contingency Table: onlyfactive judgements Tagger M YesNo TaggerAYes n yy =181 n yn =25 No n ny =12 n nn =252 Agreement for onlyfactive judgments Tagger M YesNo TaggerAYes n yy =181 n yn =8 No n ny =11 n nn =224 Agreement for onlyfactive judgments, borderline cases removed

July 5, 2003SIGdial Onlyfactive Agreement All Expressions Borderline Removed KappaagreeKappaagree % removed A & M A & S M & S

July 5, 2003SIGdial Subjective Sentence Agreement All Expressions Borderline Removed KappaagreeKappaagree % removed A & M A & S M & S

July 5, 2003SIGdial Why we exclude minor on annotations This is the first time that the Cuban government has expressed its position over the U.S. holding of the Taliban and al-Qaeda prisoners in its territory. (writer, Cuban gov) onlyfactive=no Objective Sentence!