1 Predicting Answer Location Using Shallow Semantic Analogical Reasoning in a Factoid Question Answering System Hapnes Toba, Mirna Adriani, and Ruli Manurung Faculty of Computer Science Universitas Indonesia

2 What is QAS Question answering sustem (QAS): –Input: a natural language question –Output: single answer

What is Factoid QAS Factoid QAS: –Input: an open-domin fact-based question –Output: answer –E.q: Question: –“Where was an Oviraptor fossil sitting on a nest discovered?” Answer: –‘Mongolia’s Gobi Desert’ 3

A Typical pipeline architecture Factoid QAS 4 Question analysis Query formulation Information retrieval Answer selection

A Typical pipeline architecture Factoid QAS Question analysis Determine the type of a given question, which in turn provides the expected answer type (EAT) –E.q. : person, organization, location. –named-entity recognizer (NER) is usually judged EAT 5

Semantic Analogical Reasoning (SAR) SAR predict the location of the final answer in a textual passage by employing the analogical reasoning (AR) framework from Silva et al. (2010). Author hypothesize that similar questions give similar answers. 6

Figure 1: Idea of Semantic Analogical Reasoning 7

SAR System Architecture 8

Semantic Analogical Reasoning (SAR) 9

Analogical Reasoning (AR) AR focus on the similarity between functions that map pairs to links. 10

Analogical Reasoning (AR) L ij ∈ {0, 1} : –indicator of the existence of a relation between two related objects i and j. Consider then that we also have K-dimensional vectors, each consisting of features which relates the objects i and j : = Θ [Θ 1... Θ k ] T. –This vector will represent the presence or absence of relation between two particular objects. 11

Analogical Reasoning (AR) Given the vectors of features Θ, the strength of the relation between two objects i and j is computed by performing logistic regression estimation as follows: P(L ij |x ij, Θ) = logistic(Θ T X ij ) where logistic(x) is defined as: 1 / (1 + e -x ) 12

Analogical Reasoning (AR) During AR training phase, the framework learns the weight (prior) for each feature by performing the following equation: 13

Analogical Reasoning (AR) During the AR retrieval phase, a final score that indicates the rank of predicted relations between two new objects i and j (query) and the related objects that have been learnt in a given set S is compute as follows: 14

Analogical Reasoning (AR) 15

Experiments and Evaluation objectives of experiments find the importance level of the feature set evaluate the potential of our approach to locate factoid answers in snippets and document retrieval scenarios without using any NER-tool –For this objective we run two kinds of experiments. 16

Experiments and Evaluation use the question answer pairs from CLEF 1 English monolingual of the year 2006, 2007 and training data2007 and factoid question answer pairs Testing data factoid questions

Experiments and Evaluation 18 Importance of feature

Experiments and Evaluation Gold Standard Snippets Assume: – IR process performed perfectly and returns the best snippet which covers the final answer. 19

Experiments and Evaluation 20 Gold Standard Snippets

Experiments and Evaluation Gold Standard Snippets: improve TIME and MEASURE –TIME: dd/mm/yy, dd-mmmyy, a single year number hh:mm a.m./p.m. –sometimes the chunker recognizes variations as numbers or as nouns. –MEASURE : A measurement can be written as numbers (for example: “40”) or as text (“forty”) 21

Experiments and Evaluation 22 Gold Standard Snippets ADVP = Adverb phrase NP = Noun phrase PP = Prepositional phrase O = Begin/End of a sentence or a coordinating conjinction

Experiments and Evaluation Indri Document Retrieval In the real situation, we will not have any information about the semantic chunk of the final answer. We assume that the best pair (i.e. the top-1 pair after the re-ranking process) of the AR answer features will supply us with that information. 23

Experiments and Evaluation Indri Document Retrieval performed IR process by using Indri Search Engine to retrieve the top-5 documents and pass them on to Open Ephyra and our system. Use same AR feature set as in the first experiment only use the question feature set Due to the lack of the answer features, we need to adjust the way of the re-ranking process. 24

Experiments and Evaluation 25 Indri Document Retrieval

Experiments and Evaluation 26 Indri Document Retrieval ADVP = Adverb phrase NP = Noun phrase PP = Prepositional phrase O = Begin/End of a sentence or a coordinating conjinction

Experiments and Evaluation 27 Indri Document Retrieval

Experiments and Evaluation 28 Indri Document Retrieval

Conclusion In this paper we have shown that by learning analogical linkages of question-answer pairs we can predict the location of factoid answers of a given snippet or document. Author approach achieves a very good accuracy in the OTHER answer-type 29

Experiments and Evaluation Gold Standard Snippets Compete: –Open Ephyra (Schlaefer et al., 2006) model-based NER (OpenNLP and Stanford NER) dictionary-based NER (that was specially design for TREC-QA competition) 30

Author classify the error types of our approach in three groups –(1) not covered by Indri retrieval –(2) decreasing rank of relevant document because of the AR re-ranking score function –(3) irrelevant example from the best AR pair. 31