Multi-Perspective Question Answering Using the OpQA Corpus Veselin Stoyanov Claire Cardie Janyce Wiebe Cornell University University of Pittsburgh

10/08/05HLT/EMNLP Multi-Perspective Question Answering Fact-based question answering (QA): When is the first day of spring? Do Lipton employees take coffee breaks? Vs Multi-perspective question answering (MPQA). How does the US regard the terrorist attacks in Iraq? Is Derek Jeter a bum?

10/08/05HLT/EMNLP Properties of Opinion vs. Fact answers –OpQA corpus –Traditional fact-based QA systems –Different properties of opinion questions Using fine-grained opinion information for MPQA –Annotation framework and automatic classifiers –QA experiments Talk Outline

10/08/05HLT/EMNLP Opinion Question & Answer (OpQA) Corpus 98 documents manually tagged for opinions (from the NRRC MPQA corpus [Wilson and Wiebe 2003]) 30 questions –15 fact –15 opinion [Stoyanov, Cardie, Litman, and Wiebe 2004]

10/08/05HLT/EMNLP OpQA corpus: Answer Annotations Two annotators Include every text segment contributing to an answer –Partial answers: When was the Kyoto protocol ratified? –… before May 2003 … Are the Japanese unanimous in their support of Koizumi? –… most Japanese support their prime minister … Minimum spans

10/08/05HLT/EMNLP Traditional Fact-based QA systems IR subsystem 1.Frag Frag Frag Frag Frag Frag Frag Frag 213 Linguistic filters Guesses: 1.Frag Frag ….. Documents (document fragments) Questions Syntactic filters Semantic filters

10/08/05HLT/EMNLP Characteristics of Opinion vs. Fact Answers Answer length –Syntactic and semantic class –Additional processing difficulties Partial answers –Answer generator Number of answers Length (tokens)Number of partials Fact (9.68%) Opinion (37.11%)

10/08/05HLT/EMNLP Fine-grained Opinion Information for MPQA Recent interest in the area of automatic opinion information extraction. –E.g. [Bethard, Yu, Thornton, Hatzivassiloglou, and Jurafsky 2004], [Pang and Lee 2004], [Riloff and Wiebe 2003], [Wiebe and Riloff 2005], [Wilson, Wiebe, and Hwa 2004], [Yu and Hatzivassiloglou 2003] In our evaluation: –Opinion annotation framework –Sentence-level automatic opinion classifiers –Subjectivity filters –Source filter

10/08/05HLT/EMNLP Described in [Wiebe, Wilson, and Cardie 2002] Accounts for both: –Explicitly stated opinions Joe believes that Sue dislikes the Red Sox. –Indirectly expressed opinions The aim of the report is to tarnish China’s image. Attributes include strength and source. Manual sentence-level classification –sentence subjective if it contains one or more opinions of strength >= medium Opinion Annotation Framework Described in [Wiebe, Wilson, and Cardie 2002] Accounts for both: –Explicitly stated opinions Joe believes that Sue dislikes the Red Sox. –Indirectly expressed opinions The aim of the report is to tarnish China’s image. Attributes include strength and source. Manual sentence-level classification –sentence subjective if it contains one or more opinions of strength >= medium

10/08/05HLT/EMNLP Automatic Opinion Classifiers Two sentence-level opinion classifiers from Wiebe and Riloff [2005] used for evaluation Both classifiers use unannotated data –Rulebased: Extraction patterns bootstrapped using word lists –NaiveBayes: Trained on data obtained from Rulebased PrecisionRecallF Rulebased NaiveBayes

10/08/05HLT/EMNLP Subjectivity Filters IR subsystem 1.Sent Sent Sent Sent Sent 324(o) 2.Sent 111(f) 3.Sent 431(f) 4.Sent 213(o) Subjectivity filters Document Sentences Opinion Questions Guesses 1.Sent Sent ….. Manual Rulebased NaiveBayes Baseline

10/08/05HLT/EMNLP Subjectivity Filters Cont’d Look for the rank of the first guess containing an answer Compute: 1.Sent Sent Sent 007 (ans) 4.Sent Sent 211 (ans) 6. … –Mean Reciprocal Rank (MRR) across the top 5 answers MRR = mean all_questions (1/Rank_of_first_answer) –Mean Rank of the First Answer MRFA = mean all_questions (Rank_of_first_answer)

10/08/05HLT/EMNLP Subjectivity Filters Results

10/08/05HLT/EMNLP Source Filter Manually identify the sources in the opinion questions Does France approve of the war in Iraq? Retains only sentences that contain opinions with sources matching sources in the question France has voiced some concerns with the situation.

10/08/05HLT/EMNLP Source Filter Results Performs well on the hardest questions in the corpus All questions answered within the first 25 sentences with one exception. MRRMRFA Baseline Source

10/08/05HLT/EMNLP Summary Properties of opinion vs. fact answers –Traditional architectures unlikely to be effective Use of fine-grained opinion information for MPQA –MPQA can benefit from fine-grained perspective information

10/08/05HLT/EMNLP Future Work Create summaries of all opinions in a document using fine-grained opinion information Methods used will be directly applicable to MPQA

10/08/05HLT/EMNLP Thank you. Questions?

10/08/05HLT/EMNLP Did something surprising happen when Chavez regained power in Venezuela after he was removed by a coup? What did South Africa want Mugabe to do after the 2002 elections? What’s Mugabe’s opinion about the West’s attitude and actions towards the 2002 Zimbabwe election?

10/08/05HLT/EMNLP Characteristics of Fact vs. Opinion Answers Cont’d Syntactic Constituent of the answers FactOpinion Answers in best matching category 31%16% Syntactic type of best match Verb Phrase 02 Noun Phrase 94 Clause26

10/08/05HLT/EMNLP All improvement significant using Wilcoxon Matched-Pairs Signed-Ranks Test (p<=0.05) except for source filter (p=0.81)