1. Multi-Perspective Question Answering Using the OpQA Corpus Veselin Stoyanov Claire Cardie Janyce Wiebe Cornell University University of Pittsburgh

2. 10/08/05HLT/EMNLP 2005.2 Multi-Perspective Question Answering Fact-based question answering (QA): When is the first day of spring? Do Lipton employees take coffee breaks? Vs Multi-perspective question answering (MPQA). How does the US regard the terrorist attacks in Iraq? Is Derek Jeter a bum?

3. 10/08/05HLT/EMNLP 2005.3 Properties of Opinion vs. Fact answers –OpQA corpus –Traditional fact-based QA systems –Different properties of opinion questions Using fine-grained opinion information for MPQA –Annotation framework and automatic classifiers –QA experiments Talk Outline

4. 10/08/05HLT/EMNLP 2005.4 Opinion Question & Answer (OpQA) Corpus 98 documents manually tagged for opinions (from the NRRC MPQA corpus [Wilson and Wiebe 2003]) 30 questions –15 fact –15 opinion [Stoyanov, Cardie, Litman, and Wiebe 2004]

5. 10/08/05HLT/EMNLP 2005.5 OpQA corpus: Answer Annotations Two annotators Include every text segment contributing to an answer –Partial answers: When was the Kyoto protocol ratified? –… before May 2003 … Are the Japanese unanimous in their support of Koizumi? –… most Japanese support their prime minister … Minimum spans

6. 10/08/05HLT/EMNLP 2005.6 Traditional Fact-based QA systems IR subsystem 1.Frag 324 2.Frag 111 3.Frag 431 4.Frag 213 1.Frag 324 2.Frag 111 3.Frag 431 4.Frag 213 Linguistic filters Guesses: 1.Frag 324 2.Frag 213 3. ….. Documents (document fragments) Questions Syntactic filters Semantic filters

7. 10/08/05HLT/EMNLP 2005.7 Characteristics of Opinion vs. Fact Answers Answer length –Syntactic and semantic class –Additional processing difficulties Partial answers –Answer generator Number of answers Length (tokens)Number of partials Fact 1245.1212 (9.68%) Opinion 4159.24154 (37.11%)

8. 10/08/05HLT/EMNLP 2005.8 Fine-grained Opinion Information for MPQA Recent interest in the area of automatic opinion information extraction. –E.g. [Bethard, Yu, Thornton, Hatzivassiloglou, and Jurafsky 2004], [Pang and Lee 2004], [Riloff and Wiebe 2003], [Wiebe and Riloff 2005], [Wilson, Wiebe, and Hwa 2004], [Yu and Hatzivassiloglou 2003] In our evaluation: –Opinion annotation framework –Sentence-level automatic opinion classifiers –Subjectivity filters –Source filter

9. 10/08/05HLT/EMNLP 2005.9 Described in [Wiebe, Wilson, and Cardie 2002] Accounts for both: –Explicitly stated opinions Joe believes that Sue dislikes the Red Sox. –Indirectly expressed opinions The aim of the report is to tarnish China’s image. Attributes include strength and source. Manual sentence-level classification –sentence subjective if it contains one or more opinions of strength >= medium Opinion Annotation Framework Described in [Wiebe, Wilson, and Cardie 2002] Accounts for both: –Explicitly stated opinions Joe believes that Sue dislikes the Red Sox. –Indirectly expressed opinions The aim of the report is to tarnish China’s image. Attributes include strength and source. Manual sentence-level classification –sentence subjective if it contains one or more opinions of strength >= medium

10. 10/08/05HLT/EMNLP 2005.10 Automatic Opinion Classifiers Two sentence-level opinion classifiers from Wiebe and Riloff [2005] used for evaluation Both classifiers use unannotated data –Rulebased: Extraction patterns bootstrapped using word lists –NaiveBayes: Trained on data obtained from Rulebased PrecisionRecallF Rulebased90.434.246.6 NaiveBayes79.470.674.7

11. 10/08/05HLT/EMNLP 2005.11 Subjectivity Filters IR subsystem 1.Sent 324 2.Sent 111 3.Sent 431 4.Sent 213 1.Sent 324(o) 2.Sent 111(f) 3.Sent 431(f) 4.Sent 213(o) Subjectivity filters Document Sentences Opinion Questions Guesses 1.Sent 324 2.Sent 213 3. ….. Manual Rulebased NaiveBayes Baseline

12. 10/08/05HLT/EMNLP 2005.12 Subjectivity Filters Cont’d Look for the rank of the first guess containing an answer Compute: 1.Sent 324 2.Sent 213 3.Sent 007 (ans) 4.Sent 212 5.Sent 211 (ans) 6. … –Mean Reciprocal Rank (MRR) across the top 5 answers MRR = mean all_questions (1/Rank_of_first_answer) –Mean Rank of the First Answer MRFA = mean all_questions (Rank_of_first_answer)

13. 10/08/05HLT/EMNLP 2005.13 Subjectivity Filters Results 0.4214

14. 10/08/05HLT/EMNLP 2005.14 Source Filter Manually identify the sources in the opinion questions Does France approve of the war in Iraq? Retains only sentences that contain opinions with sources matching sources in the question France has voiced some concerns with the situation.

15. 10/08/05HLT/EMNLP 2005.15 Source Filter Results Performs well on the hardest questions in the corpus All questions answered within the first 25 sentences with one exception. MRRMRFA Baseline0.491161.33 Source0.463311.27

16. 10/08/05HLT/EMNLP 2005.16 Summary Properties of opinion vs. fact answers –Traditional architectures unlikely to be effective Use of fine-grained opinion information for MPQA –MPQA can benefit from fine-grained perspective information

17. 10/08/05HLT/EMNLP 2005.17 Future Work Create summaries of all opinions in a document using fine-grained opinion information Methods used will be directly applicable to MPQA

18. 10/08/05HLT/EMNLP 2005.18 Thank you. Questions?

19. 10/08/05HLT/EMNLP 2005.19 Did something surprising happen when Chavez regained power in Venezuela after he was removed by a coup? What did South Africa want Mugabe to do after the 2002 elections? What’s Mugabe’s opinion about the West’s attitude and actions towards the 2002 Zimbabwe election?

20. 10/08/05HLT/EMNLP 2005.20 Characteristics of Fact vs. Opinion Answers Cont’d Syntactic Constituent of the answers FactOpinion Answers in best matching category 31%16% Syntactic type of best match Verb Phrase 02 Noun Phrase 94 Clause26

21. 10/08/05HLT/EMNLP 2005.21 All improvement significant using Wilcoxon Matched-Pairs Signed-Ranks Test (p<=0.05) except for source filter (p=0.81)