Jan Wiebe University of Pittsburgh Claire Cardie Cornell University Ellen Riloff University of Utah Opinions in Question Answering

Overview Techniques and tools to support multi- perspective question answering (MPQA) Goals:  produce high-level summaries of opinions  incorporate rich information about opinions extracted from text

Overview Opinion-oriented information extraction Extract opinion frames for individual expressions Combine to create opinion-oriented “scenario” templates Opinion Summary Template

MPQA Corpus Grew out of the 2002 ARDA NRRC Workshop on Multi-Perspective Question Answering Detailed annotations of opinions Freely available (thanks to David Day): nrrc.mitre.org/NRRC/publications.htm

Collaborations Interactions with end-to-end system teams Integrated corpus annotation Pilot opinion evaluation

Outline Recent activities  Subjective sentence identifier  Clause intensity identifier  Extended annotation scheme Version 1  Q&A corpus  Nested opinions  Opinion summaries What’s next

Subjective Sentence Identifier Input is unlabeled data Evaluated on manual annotations of the MPQA corpus Accuracy as good as supervised systems which classify all sentences

Subjective Sentence Identifier Bootstraps from a known subjective vocabulary, labeling the sentences it can with confidence Extraction pattern learner finds clues of subjectivity in that corpus Incorporated into a statistical model trained on the automatically labeled data Multiple classification strategies  76% accuracy with 54% baseline  80% subj. precision and 66% subj. recall  80% obj. precision: and 51% obj. recall

Clause-level intensity (strength) identification Maximum intensity of the opinions in a clause Neutral, low, medium, high Evaluated on manual annotations of the MPQA corpus

I am furious that my landlord refused to return my security deposit until I sued them. Example return my that am them sued I to refused landlord furious I until deposit securitymy High Strength Medium Strength Neutral Opinionated Sentence

Clause-level intensity (strength) identification Classification and regression learners Accuracy: how many clauses are assigned exactly the correct class? Mean Squared Error: how close are the answers to the right ones? Accuracy: classification > regression  23-79% over baseline MSE: regression > classification  57-64% over baseline

Opinion Frames direct subjective annotation Span: “strongly criticized and condemned” Source: Strength (intensity): high Attitudes: negative toward the report Target: report The report has been strongly criticized and condemned by many countries.

Major Attitude Types Positive Negative Arguing for ones world view Intention

Negative and Positive Example People are happy because Chavez has fallen, she said. direct subjective annotation span: are happy source: attitude: attitude annotation span: are happy because Chavez has fallen type: positive and negative positive target: negative target: target annotation span: Chavez has fallen target annotation span: Chavez

Arguing for World View Example Putin remarked that events in Chechnia “could be interpreted only in the context of the struggle against international terrorism.” direct subjective annotation span: remarked source: attitude: attitude annotation span: could be interpreted only in the context of the struggle against international terrorism type: argue for world view target: target annotation span: events in Chechnia

Characteristics Sarcastic "Great, keep on buying dollars so there'll be more and more poor people in the country," shouted one. Speculative Leaders probably held their breath… Characteristics of the linguistic realization

Q&A Corpus Includes 98 documents from the NRRC corpus, split into four topics: Kyoto Protocol 2002 elections in Zimbabwe U.S. annual human rights report 2002 coup in Venezuela

Q&A Corpus Includes 30 questions 15 questions classified as fact  What is the Kyoto Protocol about?  What is the Kiko Network?  Where did Mugabe vote in the 2002 presidential election? 15 questions classified as opinion  How do European Union countries feel about the US opposition the Kyoto protocol?  Are the Japanese unanimous in their opinion of Bush’s position on the Kyoto Protocol?  What was the American and British reaction to the reelection of Mugabe?

Q&A Corpus Answers annotations added by two annotators  Minimal spans that constituted or contributed to an answer Confidence Partial?

Difficulties in Corpus Creation Annotating answers  Difficult to decide what constitutes an answer: Q: “Did most Venezuelans support the 2002 coup?” A: “Protesters…failed to gain the support of the army.” ???  Not clear what sources to attribute to collective entities European Union: The EU Parliament? Great Britain? GB government? Tony Blair? The Japanese: The Japanese government? Emperor Akihito? Empress Michiko? The Kiko Network?

Q&A Corpus Interannotator agreement  85% on average  using Wiebe et. al’s agr(a||b) measure  78% and 93%, respectively for each annotator

Evaluating MPQA Opinion Annotations Answer probability: estimate P(opinion answer | opinion question) P(fact answer | fact question)  Low-level opinion information reliable predictor facts: 78% opinions: 93% Answer rank  Sentence-based retrieval  Filter based on opinion annotations  Examine rank of first sentence w/answer  Filtering improves answer rank

Summary Representations of Opinions Direct subjective annotation Source: Attitude: Opinion Summary Template

Reporting in text Clapp sums up the environmental movement’s reaction: “The polluters are unreasonable’’ Charlie was angry at Alice’s claim that Bob was unhappy

Hierarchy of Perspective & Speech Expressions Charlie was angry at Alice’s claim that Bob was unhappy angryclaimimplicit speech event unhappy sums up implicit speech event reaction Clapp sums up the environmental movement’s reaction: “The polluters are unreasonable’’

Baseline 1: Only filter through writer 66% correct angryclaim implicit unhappy

Baseline 2: Dependency Tree 72% correct angry implicit claim unhappy claim unhappy claim unhappy

ML Approach Features  Parse-based  Positional  Lexical  Genre-specific IND decision trees (mml criterion) 78% correct

Summary Representations of Opinions Direct subjective annotation Source: Attitude: Opinion Summary Template

Opinion Summaries Summaries based on manual annotations  Single-document summaries  Opinion annotations grouped by source and target  Sources characterized by degree of subjectivity/objectivity  Simple graph-based graphical interface Overview of entire graph Focus on portion of the graph Drill-down to opinion annotations (highlighted) Grouping/deleting of sources/targets JGRAPH package

The next 6 months Identify individual expressions of subjectivity Perform manual annotations Extract Sources Opinion summaries with automatic annotations