Machine Reading

Introduction Definition: a task that deals with the automatic understanding of texts the “KnowItAll” research group at the University of Washington Target: understanding text

Related work Question Answering Information Extraction utilize supervised learning techniques, which rely on hand-tagged training examples Information Extraction Often utilizes extraction rules learned from example extractions of each target relation Impractical to generate a set of hand-tagged examples

Difference Fact MR the relations are not known in advance impractical to generate hand-tagged examples of each relation of interest. MR Inherently unsupervised forging and updating connections between beliefs vs. focusing on isolated “nuggets” obtained from text

Reference [1] Etzioni, Oren, Michele Banko, and Michael J. Cafarella. "Machine Reading."AAAI. Vol. 6. 2006.

QA4MRE In 2011, 2012, 2013

Summary Main objective Develop a methodology for evaluating Machine Reading systems through Question Answering and Reading Comprehension Tests. Systems should be able to extract knowledge from large volumes of text and use this knowledge to answer questions. The methodology should allow the comparison of systems' performance and the study of the best approaches.

QA4MRE Organization, tasks, participants, results, winner’s methods ..

CLEF 2011 Host Time Place the University of Amsterdam The Netherlands  19-22 September 2011 Place Amsterdam, the Kingdom of the Netherlands

CLEF 2012 Host Time Place the University "La Sapienza"  17-20 September 2012  Place Rome, Italy

CLEF 2013 Host Time Place the Technical University of Valencia  23-26 September 2013   Place Spain

The Task Reading of single documents and the identification of the answers to a set of questions In the form of multiple choice Only one correct answer Require both semantic understanding and reasoning process

Requirements Understand the test questions Analyze the relation among entities contained in questions and entities expressed by the candidate answers Understand the information contained in the documents Extract useful pieces of knowledge from the background collections Select the correct answer from the five alternatives proposed.

Testing Data Available in several languages Arabic, Bulgarian, English, German, Italian, Romanian, and Spanish Test sets were divided into topics AIDS, Climate Change, Music, Society and Alzheimer’s disease Background collection, testing documents, candidate answers are provided

Background Collections 2011 2012&2013

Evaluation In order to improve results, systems might reduce the amount of incorrect answers while keeping the proportion of correct ones, by leaving some questions unanswered consider the possibility of leaving questions unanswered Responses: R W NoA NoA_R ( with a right hypothetical answer ) NoA_W ( with a wrong hypothetical answer )

Difference In 2013 Systems able to decide whether all candidate answers were incorrect or not should be rewarded over systems that just rank answers Introducing in our tests a portion of questions (39%) where none of the options are correct and including a new last option in all questions: “None of the above answers is correct” (NCA) (baseline should be 0.39) Obviously the answer NCA is better than NoA

Evaluation Measure nR: number of questions correctly answered. nU: number of questions unanswered. n: total number of questions Baseline:0.2

Evaluation Measure Secondary measure nR: number of questions correctly answered. nU: number of questions unanswered. n: total number of questions Baseline:0.2

Evaluation Measure Evaluate the validation performance nUR: number of unanswered questions whose candidate answer was correct nUW: number of unanswered questions whose candidate answer was incorrect nUE: number of unanswered questions whose candidate answer was empty

Overview of results

Results in 2011

Result in 2012

Result in 2013

Result in 2013

The Use of External Knowledge Run 1 : no external knowledge allowed but the background collections provided by the organization

Websites http://clef2011.clef-initiative.eu/index.php http://clef2012.clef-initiative.eu/index.php http://clef2013.clef-initiative.eu/index.php

Thank you