Techniques Used in Modern Question-Answering Systems Candidacy Exam Elena Filatova December 11, 2002 Committee Luis GravanoColumbia University Vasileios Hatzivassiloglou Department of Computer Science Rebecca J. Passonneau

Present vs Past Research on QA Current systems –Mainly systems written for TREC conference factoid questions short answers huge text collections Related systems –IR queries vs questions return documents vs short answers –Systems based on semantic representations ( Lehnert ): questions about one text vs text collections inference from semantic structure of a text vs searching for an answer in the text –One type of output (NP) from a closed collection ( Kupiec ) answer inference vs answer extraction

Lehner’t system John loved Mary but she didn’t want to marry him. One day, a dragon stole Mary from the castle. John got on top of his horse and killed the dragon. Mary agreed to marry him. They lived happily ever after. Q: Why did Mary agree to marry John? A: Because she was indebted to him Problems stated: –right classification –dependency of answer inference procedure on the type of the question

Current QA Systems question analysis question query extracted document s rules for answer list of answers Information Extraction right query long text domain dependency predefined types of answers Information Retrieval

Plan Classification Information (document) retrieval –Query formation Information extraction –Passage extraction –Answer extraction Usage of answer redundancy on Web in QA QA for restricted domain Evaluation procedure for current QA systems and analysis of the performance

Classification and QA question analysis question query extracted document s rules for answer list of answers

Theory of Classification Rosch et al: classification of basic objects World is structured: real-world attributes do not occur independently of each other: object_has(wings) => P(object_has(feathers)) > P(object_has(fur)) Each category (class) – set of attributes that are common for all the objects in the category Types of categories Superordinate – small amount of common attributes ( furniture ) Subordinate – a lot of common attributes ( floor lamp, desk lamp ) Basic – optimal amount of common attributes ( lamp ): basic objects are the most inclusive categories which delineate the correlation structure of the environment Though c lassification is a converging problem for objects, it is not possible to compile a list of all possible basic categories.

QA classification. Hierarchical/nonhierarchical classification –Even if there exist hierarchy in the classification it can be represented as flat: detailed classes + other class Amount of types (MULDER – 3 types vs Webclopedia – over 140 types) Trade off between –Detailed classes for better answer extraction and –High precision in defining the classes Usage of semantics Usage of syntax –Most of syntactic parsers are built on corpora which do no contain a lot of questions (WSJ) => need of additional corpus Attempts to automate this process –Maximum Entropy ( Ittycheriah ) –Classifiers ( Li&Roth )

Why QA classification is important? Usage of question type for 1.query construction question keywords + filtering mechanism ( Harabagiu ) synonyms and syn.sets from WordNet ( Webclopedia ) in both cases there is no connection with possible answer space information retrieval ( Agichtein, Berger ) there is connection between question and answer spaces but these types do not give the type of the answer 2. searching for a correct answer in the passage extracted from a text

Logical Forms Syntactic analysis plus semantic => logical form Mapping of question and potential answer LFs to find the best match ( Harabagiu, Webclopedia )

Query formation WordNet: synonyms, hyponyms, etc. Morphology: verbal forms, plural/single nouns, etc. Knowledge of the domain (IBM’s system) Statistical methods for connecting question and answer spaces: –Agichtein : automatic acquisition of patterns that might be good candidates for query expansion 4 ‘types‘ of question –Berger : to facilitate query modification (expansion) each question term gets a set of answer terms FQA: closed set of question-answer pairs

Information retrieval Classical IR is the first step of QA Vector-space model (calculation of similarity between terms in the query and terms in the document) IR techniques used in current QA systems are usually for one database (either web or TREC collection) Is it possible to apply Distributed IR techniques? –domain restricted QA with extra knowledge about the text collection IBM system –“splitting” one big collection of documents into smaller collections about specific topics –it might require change in classification: type of the question might cause the changes in query formulation, document extraction process, answer extraction process

question analysis question query extracted document s rules for answer list of answers Information Extraction Information Retrieval

Passage extraction Passages of particular length ( Cardie ) + Vector representation for each passage Paragraphs or sentences Classical text excerpting –Each sentence is assigned a score –Retrieved passages are formed by taking the sentences with the highest score Global-Local Processing ( Salton ) McCallum : passage extraction based not only on words but also on other features (e.g. syntactic constructions)

Information Extraction Domain dependency ( Grishman ) predefined set of attributes for the search specific for each topic, e.g. terrorism: victims, locations, perpetrators usually a lot of manually tagged data for training or texts divided into two groups: one topic – all other texts ( Riloff ) in both cases division into topics is a necessary step which is not applicable to open domain QA systems

What information can be extracted (IE) Named entities (NE-tagging) –Numbers (incl. dates, ZIP codes, etc.) –Proper names (locations, people, etc.) –Other depending on the system TREC8 – 80% questions asked for NEs NEs might also support Correlated entity: mini-CV ( Srihari ) Who is Julian Hill? name; age; gender; position; affiliation; education General events ( Srihari ) Who did what to whom when More complicated IE techniques lead QA back to AI approach

Answer Extraction Three main techniques for answer extraction are based on: 1.syntactic-semantic tree dependencies: ( Harabagiu, Webclopedia ) LF of the question is mapped to LF of possible answers 2.surface patterns ( Webclopedia ) – ( -) – was born on Good patterns require detailed classification: NUMBER vs DOB 3.text window –Cardie: query-dependant text summarization of text passages with/without syntactic and semantic information LF mapping classical MT surface patterns example-based MT text window statistical MT

Usage of Web (Answer redundancy) Multiple formulation of answer can useful for: 1.IR stage: increased chances to find an answer that matches query ( Clarke, Brill ) no need in searching for an exact formulation of the answer 2. IE stage: facilitation of answer extraction ( Agichtein, Ravichandran, Brill ) create a list of patterns which might contain the answer either completely automatic (Agichtein) or using handwritten filters based on question types and domain (Brill) 3.Answer validation ( Magnini ) correct answer redundancy

Domain restricted applications FAQ (different from IR or QA) –match the input question with a list of already existing questions –predefined output (according to the above question matching) Rillof –5 types of questions –answer extraction from a given text => no IR stage –always there is an answer (unique answer) IBM system –based on good knowledge of inner structure of IBM web- site –Use of FAQ techniques results are better than for open-domain QA systems restricted-domain MT vs open-domain MT

Evaluation IR and IE have different evaluation measures –IR: each document is marked either relevant/non-relevant  recall + precision –IE: gold standard answer key enumerates all acceptable responses  recall + precision –QA: mean reciprocal rank (MRR)  For each question: receive score equal to reciprocal of rank of first correct response, or 0 if no correct response found. Overall system score is mean of individual question scores. N – amount of questions asked; Ki = rank of the correct answer or 0; RAR =1/ Ki

Future of QA FROMTO Questions: Simple facts Questions: Complex: Uses Judgments Terms; Knowledge of User Context Needed Answers: Simple Factoid Answers found in Single Document Answers: Search Mult. Sources; Fusion of Info; Resolution of Conflicting Data; Interpretations, Conclusions