Evaluating Question Answering Validation Anselmo Peñas (and Alvaro Rodrigo) NLP & IR group UNED nlp.uned.es Information Science Institute Marina del Rey, December 11, 2009

UNED nlp.uned.es Old friends Question Answering Nothing else than answering a question Natural Language Understanding Something there, if you are able to answer a question QA: extrinsic evaluation for NLU Suddenly… (See the track?) …The QA Track at TREC

UNED nlp.uned.es Question Answering at TREC Object of evaluation itself Redefined as a (roughly speaking): Highly-precision-oriented IR task Where NLP was necessary Specially for Answer Extraction

UNED nlp.uned.es What’s this story about? QA Tasks at CLEF Multiple Language QA Main TaskResPubliQA Temporal restrictions and lists Answer Validation Exercise (AVE) GikiCLEF Real Time QA over Speech Transcriptions (QAST) WiQA WSD QA

UNED nlp.uned.es Outline 1. Motivation and goals 2. Definition and general framework 3. AVE AVE 2007 & QA 2009

UNED nlp.uned.es Short cycleLong cycle Out-line 1. Analysis of current systems performance 2. Mid term goals and strategy 3. Evaluation Task definition 4. Analysis of the evaluation cycle Result analysis Methodology analysis Generation of methodology and evaluation resources Task activation and development

UNED nlp.uned.es Systems performance (Spanish) Overall Best result <60% Definitions Best result >80% NOT IR approach

UNED nlp.uned.es Pipeline Upper Bounds SOMETHING to break the pipeline Question Answer Question analysis Passage Retrieval Answer Extraction Answer Ranking xx= Not enough evidence

UNED nlp.uned.es Results in CLEF-QA 2006 (Spanish) Perfect combination 81% Best system 52,5% Best with ORGANIZATION Best with PERSON Best with TIME

UNED nlp.uned.es Collaborative architectures Different systems response better different types of questions Specialization Collaboration QA sys 1 QA sys 2 QA sys 3 QA sys n Question Candidate answers SOMETHING for combining / selecting Answer

UNED nlp.uned.es Collaborative architectures How to select the good answer? Redundancy Voting Confidence score Performance history Why not deeper content analysis?

UNED nlp.uned.es Mid Term Goal Goal Improve QA systems performance New mid term goal Improve the devices for: Rejecting / Accepting / Selecting Answers The new task (2006) Validate the correctness of the answers Given by real QA systems......the participants at CLEF QA

UNED nlp.uned.es Outline 1. Motivation and goals 2. Definition and general framework 3. AVE AVE 2007 & QA 2009

UNED nlp.uned.es Define Answer Validation Decide whether an answer is correct or not More precisely: The Task: Given Question Answer Supporting Text Decide if the answer is correct according to the supporting text Let’s call it Answer Validation Exercise (AVE)

UNED nlp.uned.es Whish list Test collection Questions Answers Supporting Texts Human assessments Evaluation measures Participants

UNED nlp.uned.es Evaluation linked to main QA task Question Answering Track Systems’ answers Systems’ Supporting Texts Answer Validation Exercise Questions (ACCEPT / REJECT) Human Judgements (R,W,X,U) QA Track results Mapping (ACCEPT / REJECT) Evaluation AVE Track results Reuse human assessments

UNED nlp.uned.es Candidate answer Supporting Text Answer is not correct or not enough evidence Question Answer is correct Answer Validation Answer Validation Exercise (AVE) AVE Textual Entailment Hypothesis Automatic Hypothesis Generation AVE 2006

UNED nlp.uned.es Outline Motivation and goals Definition and general framework AVE 2006 Underlying architecture: pipeline Evaluating the validation As RTE exercise: pairs text-hypothesis AVE 2007 & 2008 QA 2009

UNED nlp.uned.es AVE 2006: A RTE exercise If the text semantically entails the hypothesis, then the answer is expected to be correct. Question Supporting snippet Exact Answer QA system Hypothesis Text Entailment? Is this true? Yes 95% with current QA systems (J LOG COMP 2009)

UNED nlp.uned.es Collections AVE 2006 Available at:nlp.uned.es/clef-qa/ave/ Testing (pairs entail.)Training English2088 (10% YES)2870 (15% YES) Spanish2369 (28% YES)2905 (22% YES) German1443 (25% YES) French3266 (22% YES) Italian1140 (16% YES) Dutch807 (10% YES) Portuguese1324 (14% YES)

UNED nlp.uned.es Evaluating the Validation Validation Decide if each candidate answer is correct or not YES | NO Not balanced collections Approach: Detect if there is enough evidence to accept an answer Measures: Precision, recall and F over correct answers Baseline system: Accept all answers

UNED nlp.uned.es Evaluating the Validation Correct Answer Incorrect Answer Answer Accepted n CA n WA Answer Rejected n CR n WR

UNED nlp.uned.es Results AVE 2006 LanguageBaseline (F) Best system (F) Reported Techiques English.27.44Logic Spanish.45.61Logic German.39.54Lexical, Syntax, Semantics, Logic, Corpus French.37.47Overlapping, Learning Dutch.19.39Syntax, Learning Portuguese.38.35Overlapping Italian.29.41Overlapping, Learning

UNED nlp.uned.es Outline Motivation and goals Definition and general framework AVE 2006 AVE 2007 & 2008 Underlying architecture: multi-stream Quantify the potential benefit of AV in QA Evaluating the correct selection of one answer Evaluating the correct rejection of all answers QA 2009

UNED nlp.uned.es QA sys 1 QA sys 2 QA sys 3 QA sys n Question Candidate answers + Supporting Texts Answer Validation & Selection Answer Participant systems in a CLEF – QA Evaluation of Answer Validation & Selection AVE 2007 & 2008

UNED nlp.uned.es Collections What is Zanussi? was an Italian producer of home appliances Zanussi For the Polish film director, see Krzysztof Zanussi. For the hot-air balloon, see Zanussi (balloon). Zanussi was an Italian producer of home appliances that in 1984 was bought who had also been in Cassibile since August 31 Only after the signing had taken place was Giuseppe Castellano informed of the additional clauses that had been presented by general Ronald Campbell to another Italian general, Zanussi, who had also been in Cassibile since August (1985) 3 Out of 5 Live (1985) What Is This?

UNED nlp.uned.es Evaluating the Selection Goals Quantify the potential gain of Answer Validation in Question Answering Compare AV systems with QA systems Develop measures more comparable to QA accuracy

UNED nlp.uned.es Evaluating the selection Given a question with several candidate answers Two options: Selection Select an answer ≡ try to answer the question Correct selection: answer was correct Incorrect selection: answer was incorrect Rejection Reject all candidate answers ≡ leave question unanswered Correct rejection: All candidate answers were incorrect Incorrect rejection: Not all candidate answers were incorrect

UNED nlp.uned.es Evaluating the Selection n questions n= n CA + n WA + n WS + n WR + n CR Question with Correct Answer Question without Correct Answer Question Answered Correctly (One Answer Selected) n CA - Question Answered Incorrectly n WA n WS Question Unanswered (All Answers Rejected) n WR n CR

UNED nlp.uned.es Evaluating the Selection Rewards rejection (not balanced cols) Interpretation for QA: all questions correctly rejected by AV will be answered correctly

UNED nlp.uned.es Evaluating the Selection Interpretation for QA: questions correctly rejected has value as if they were answered correctly in qa_accuracy proportion

UNED nlp.uned.es Analysis and discussion (AVE 2007 Spanish) Validation Selection Comparing AV & QA

UNED nlp.uned.es Techniques in AVE 2007 Generates hypotheses 6 Wordnet 3 Chunking 3 n-grams, longest common Subsequences 5 Phrase transformations 2 NER 5 Num. Expressions 6 Temp. expressions 4 Coreference resolution 2 Dependency analysis 3 Syntactic similarity4 Functions (sub, obj, etc)3 Syntactic transformations1 Word-sense disambiguation2 Semantic parsing4 Semantic role labeling2 First order logic representation3 Theorem prover3 Semantic similarity2

UNED nlp.uned.es Conclusion of AVE Answer Validation before It was assumed as a QA module But no space for its own development The new devices should help to i mprove QA they Introduce more content analysis Use Machine Learning techniques Are able to break pipelines or combine streams Let’s transfer them to QA main task

UNED nlp.uned.es Outline Motivation and goals Definition and general framework AVE 2006 AVE 2007 & 2008 QA 2009

UNED nlp.uned.es CLEF QA 2009 campaign ResPubliQA: QA on European Legislation GikiCLEF: QA requiring geographical reasoning on Wikipedia QAST: QA on Speech Transcriptions of European Parliament Plenary sessions

UNED nlp.uned.es CLEF QA 2009 campaign Task Registered groups Participant groups Submitted Runs Organizing people ResPubliQA (baseline runs) 9 Giki CLEF27817 runs2 QAST12486 (5 subtasks)8 Total 59 showed interest 23 Groups 147 runs evaluated 19 + additional assessors

ResPubliQA 2009: QA on European Legislation Organizers Anselmo Peñas Pamela Forner Richard Sutcliffe Álvaro Rodrigo Corina Forascu Iñaki Alegria Danilo Giampiccolo Nicolas Moreau Petya Osenova Additional Assessors Fernando Luis Costa Anna Kampchen Julia Kramme Cosmina Croitoru Advisory Board Donna Harman Maarten de Rijke Dominique Laurent

UNED nlp.uned.es Evolution of the task Target languages Collections News News Wikipedia Nov European Legislation Number of questions Type of questions 200 Factoid + Temporal restrictions + Definitions - Type of question + Lists + Linked questions + Closed lists - Linked + Reason + Purpose + Procedure Supporting information DocumentSnippetParagraph Size of answer SnnipetExactParagraph

UNED nlp.uned.es Collection Subset of JRC-Acquis (10,700 docs x lang) Parallel at document level EU treaties, EU legislation, agreements and resolutions Economy, health, law, food, … Between 1950 and 2006

UNED nlp.uned.es 500 questions REASON Why did a commission expert conduct an inspection visit to Uruguay? PURPOSE/OBJECTIVE What is the overall objective of the eco-label? PROCEDURE How are stable conditions in the natural rubber trade achieved? In general, any question that can be answered in a paragraph

UNED nlp.uned.es 500 questions Also FACTOID In how many languages is the Official Journal of the Community published? DEFINITION What is meant by “whole milk”? No NIL questions

UNED nlp.uned.es Systems response No Answer ≠ Wrong Answer 1. Decide if they answer or not [ YES | NO ] Classification Problem Machine Learning, Provers, etc. Textual Entailment 2. Provide the paragraph (ID+Text) that answers the question Aim To leave a question unanswered has more value than to give a wrong answer

UNED nlp.uned.es Assessments R: The question is answered correctly W: The question is answered incorrectly NoA: The question is not answered NoA R: NoA, but the candidate answer was correct NoA W: NoA, and the candidate answer was incorrect Noa Empty: NoA and no candidate answer was given Evaluation measure: Extension of the traditional accuracy (as proportion of questions correctly answered) Considering unanswered questions

UNED nlp.uned.es Evaluation measure n: Number of questions n R : Number of correctly answered questions n U : Number of unanswered questions

UNED nlp.uned.es Evaluation measure If n U = 0 then R /n  Accuracy If n R = 0 then If n U = n then Leave a question unanswered gives value only if this avoids to return a wrong answer Accuracy The added value is the performance shown with the answered questions: Accuracy

UNED nlp.uned.es List of Participants SystemTeam elixELHUYAR-IXA, SPAIN iciaRACAI, ROMANIA iiitSearch & Info Extraction Lab, INDIA ilesLIMSI-CNRS-2, FRANCE isikISI-Kolkata, INDIA logaU.Koblenz-Landau, GERMAN miraMIRACLE, SPAIN nlelU. politecnica Valencia, SPAIN synaSynapse Developpment, FRANCE uaicAI.I.Cuza U. of IASI, ROMANIA unedUNED, SPAIN

UNED nlp.uned.es Value of reducing wrong answers R #NoA W #NoA empty combination icia092roro icia091roro UAIC092roro UAIC091roro base092roro base091roro

UNED nlp.uned.es Detecting wrong answers R #NoA W#NoA empty combination loga091dede loga092dede base092dede base091dede Maintaining the number of correct answers, the candidate answer was not correct for 83% of unanswered questions Very good step towards improving the system

UNED nlp.uned.es IR important, not enough R#NoA W#NoA empty combination uned092enen uned091enen nlel091enen uaic092enen base092enen base091enen elix092enen uaic091enen elix091enen syna091enen isik091enen iiit091enen elix092euen elix091euen Achievable Task Perfect combination is 50% better than best system Many systems under the IR baselines

UNED nlp.uned.es Outline Motivation and goals Definition and general framework AVE 2006 AVE 2007 & 2008 QA 2009 Conclusion

UNED nlp.uned.es Conclusion New QA evaluation setting Assuming that To leave a question unanswered has more value than to give a wrong answer This assumption give space to further development QA systems And hopefully improve their performance

Thanks! Acknowledgement: EU project T-CLEF (ICT )