1. Answer Validation Exercise Anselmo Peñas UNED NLP Group QA@CLEF 2005 Breakout session

2. QA System Question DOC + Answer Human assessment Correct What is BMW? Bayerische Motoren Werke

3. Answer validation What is BMW? Bayerische Motoren Werke + DOC According to DOC, Is it true the following? BMW is Bayerische Motoren Werke YES / NO Similar to a closed question: Is BMW Bayerische Motoren Werke?

4. Exercise 1. Take all answers of all runs (in the lang) 2. Reformulate as statements 3. Give them to the participant systems 4. Perform the automatic validation Is the statement true according to the document? 5. Compare the automatic validation with the human assessment

5. Why? A good answer validation can improve systems performance Example: INAOE achieves 80% acuracy in Definitions Expected improvement in systems self-scoring Better criteria for collaborative systems 40% accuracy alone versus 70% accuracy in perfect combination Feedback for a system’s component Deal with closed questions (and may be inference) Newcomers can start with a single module

6. Training data QA@CLEF 2003 2004 2005 assessments ¿Qué es BMW? BMW es Bayerische Motoren Werke un tradicional fabricante de automviles grandes Bayerische Motoren Werke incertidumbre sobre el futuro de la firma británica Ricardo Garca Galiano Luis Martnez Osella ¿Qué son las FARC?

7. QA System Question DOC + Answer Human assessment What is BMW? Uncertain about the future of the company Given DOC, Is it true the following? BMW is uncertain about the future of the company wrong But... Sure?

8. Thanks!