1. Probabilistic Model for Definitional Question Answering Kyoung-Soo Han, Young-In Song, and Hae-Chang Rim Korea University SIGIR 2006

2. INTRODUCTION Definitional question answering is a task of answering definitional questions used for finding out conceptual facts or essential events about the question target. EX: “What is X?” or “Who is X?”

3. INTRODUCTION A short passage cannot answer the definitional questions because a definition needs several essential information (information nuggets) about the target Each answer nugget is naturally represented by a short noun phrase or a verb phrase Good definitional QA systems have to find out more answer nuggets with shorter length.

4. Topic and Definition We can suppose that the following sentences are answer candidates for a question “What is NASA?”. S1: NASA is the agency responsible for the public space program of the USA. S2: NASA was established in 1958. S3: The headquarters of NASA is located in Washington, D.C. S4: NASA announced the new annual budget. S5: John who works for NASA gave a housewarming party yesterday. S6: Ji-Sung Park is a famous football player from South Korea.

5. Topic and Definition Among the candidates, the sentences representing the topic of “NASA” are {S1, S2, S3, S4}, and those representing the definition are {S1, S2, S3, S6}. {S1, S2, S3} as the answer to the question.

6. Definitional Question Answering Model the answer A to a definitional question about the target is the intersection of T and D T: The sentence S includes the contents related to the topic of the target X. D: The sentence S represents the definition. Definitional QA can be considered as a task of finding the sentences which maximize a joint probability P(T,D|S)

7. Definitional Question Answering Model P(T,D|S) = P(T|S)P(D|T,S) assuming T and D are conditionally independent given S

8. Definitional Question Answering Model It is not necessary to calculate the exact probability in order to rank the sentences. If the sentence S is a sequence of words w1w2 · · ·wn, the function is rewritten as follows:

9. Definitional Question Answering Model P(w1,n|T) is the probability that the word sequence w1,n is generated from the target topic, and we call it topic language model. P(w1,n|D) is the probability that w1,n is generated from the definition representations, and we call it definition language model. P(w1,n) is the prior probability of w1,n, and we call it general language model.

10. General Language Model P(w1,n) = P(w1)P(w2|w1)P(w3|w1,2) · · · P(wn|w1,n−1) Assuming the word occurrences are independent of one another

11. Topic Language Model We model the target topic using the following several evidences:  Top ranked documents R retrieved from the collection by the query X:  Definitions E for the target X from external resources (such as online dictionary)  Top ranked web pages W retrieved from the WWW by the query X:

12. Topic Language Model Where α + β + γ = 1 Each probability is estimated by Dirichlet smoothing

13. Definition Language Model We constructed the definition corpus by collecting the definitions for arbitrary definition targets from the online resources, and estimated the probability using the definition corpus. The word probability distribution can be different depending on the domain of the definition target. For example, “president”, “scientist”, “born”, and “died” will frequently occur in the definition for a person. On the other hand, “established”, “member”, “headquarters”, and “branch” will do in the definition for an organization.

14. Definition Language Model P(w i |D tX ) is the probability that word w i is generated from the definition corpus whose domain for definition target is equal to the domain t X for question target P(w i |D all ) is the probability that w i is generated from the definition corpus for all domains. λ is a interpolation parameter

15. Definition Language Model We used three domains in this paper: person, organization, and term

16. DEFINITIONAL QUESTION ANSWERING SYSTEM Given a definitional question, the question target is extracted and its type is identified. From a question such as “Who is Andrew Carnegie?”, for example, “Andrew Carnegie” is extracted by using simple rules. The type of the target is identified by a named entity tagger, BBN Identifinder We classify the target into three types: person, organization, and term. If a target is not classified into person or organization by the named entity tagger, it is classified into term.

17. Document Retrieval Answer candidates are extracted from the retrieved relevant documents. The document retrieval is carried out by using BM25 scoring function of OKAPI. The query consists of each word of the question target

18. Answer Candidate Extraction We try to extract target-related parts of sentences using syntactic structure of the sentences. If the parts are extracted, they are used as answer candidates. Otherwise, the sentences are used as the candidates. We extract noun and verb phrases from the sentences using the syntactic definition patterns

19. Answer Candidate Selection A = {a j |LDS(a j ) > δ sel }

20. Experiments Setup We have experimented with 50 TREC 2003 topics and 64 TREC 2004 topics, and we found the answer from the AQUAINT corpus used for TREC Question Answering track evaluation. The TREC answer set for the definitional QA task consists of several definition nuggets for each target, and each nugget is a short string. We evaluated our system using the automatic measure POURPRE. The POURPRE estimates the TREC metric, nugget recall, precision, and F-measure, using term co- occurrences between answer nugget and system output.

21. Experiments Setup For topic modeling, we collected external definitions from : Acronym Finder, Biography.com, Columbia Encyclopedia, Wikipedia, FOLDOC, The American Heritage Dictionary of the English Language, Online Medical Dictionary, and Google Glossary. We also used ten web pages retrieved by the Google search engine and five local documents retrieved from the AQUAINT corpus for topic modeling. For definition modeling, we constructed definition corpus from the above sites according to the target type: 14,904 persons, 994 organizations, and 3,639 terms entries. We processed top 200 documents retrieved in all experiments.

22. Topic Modeling The α, β, γ are weight of top ranked documents R, external definitions E, and top ranked web pages W, respectively, in equation 7 for the topic modeling. λ and μ are set to 0.6 and 2,000, respectively. δ sel is not applied

23. Topic Modeling The experimental results show:  The target topic is best represented by the external definitions. As the external definitions provide core information about the question target without noise  The external definitions are the most confident, followed by the top ranked documents and web pages, in order.

24. Definition Modeling λ is the degree to which the target type has an effect on the definition modeling.

25. Comparison with Other Systems proposed(1000,10) is the system whose target length of the answer and score threshold δ sel is set to 1,000 byte and 10, respectively.

26. CONCLUSIONS We proposed a probabilistic model for definitional QA, analyzing the problem into two main components, topic and definition. The experimental results show that the external definition which has almost no noise is the most valuable information for topic modeling