1. AQUAINT IBM PIQUANT ARDACycorp Subcontractor: PIQUANT Question Answering System ARDA AQUAINT Program June Workshop 2002 This work was supported in part by the Advanced Research and Development Activity (ARDA)'s Advanced Question Answering for Intelligence (AQUAINT) Program under contract number MDA904-01-C-0988. Dave Ferrucci, John Prager, Jennifer Chu-Carroll, Chris Welty, Chris Cesar and Scott Fahlman

2. IBM - PIQUANT IBM Research Subcontractor: Cycorp Overview Progress Update Architecture Qplans Working Example Answer Selection and Resolution Performance Improvements Summary

3. IBM - PIQUANT IBM Research Subcontractor: Cycorp PIQUANT Research Objectives Integration & impact of knowledge based system (e.g., Cyc) in QA Extensible QA architectures Declarative question plans Parallel solution paths and pervasive confidence processing Deeper linguistic & knowledge-based analysis

4. IBM - PIQUANT IBM Research Subcontractor: Cycorp Progress Since AQUAINT Kickoff Architecture Design Support for multiple answering agents, solution paths and knowledge sources Centralized ontology management & uniform access to knowledge sources New question plan modules Improved Ranking Enhanced Answer Selection using deeper linguistic analysis Integration of Cyc in Answer Resolution for “sanity checking” Integration of multiple knowledge sources Answering question previously missed Multiple solutions paths based on alternative question decomposition Integration of Cyc as a knowledge source

5. IBM - PIQUANT IBM Research Subcontractor: Cycorp Architectural Limitations as of TREC10 Pipeline Single Answering Approach Limited Extensibility Single Solution Source WordNet added as second-class citizen No Knowledge System component Limited question understanding Shallow conceptual map from Q to A Limited to explicit matches -- cut-off from inferred possibilities “Explanations” limited to text passages containing answers Can’t filter out crazy answers

6. IBM - PIQUANT IBM Research Subcontractor: Cycorp Classic Pipeline with WordNet Question Search Answer Selection Answer Answer Type HitListText Query WordNet WN Query WN Answer Answer Classification Question Analysis

7. IBM - PIQUANT IBM Research Subcontractor: Cycorp Knowledge Source Services Question WordNet Answer Resolution Answer Type HitList Text Query Search KB Query Answer Selection Answer Justification & Presentation Answers Text Search Answers Answer Classification Question Analysis Cyc Cyc Answers WordNet Answers

8. IBM - PIQUANT IBM Research Subcontractor: Cycorp Answering Agents Question WordNet Answer Resolution Answer HitList Search QGoals Answer Selection Answer Justification & Presentation Answers Cyc Convert Question to Web Query QFrame Web Complex Decomposition & Planning Answering Agents KS Adaptation Layer Answer Classification Question Analysis Causality

9. IBM - PIQUANT IBM Research Subcontractor: Cycorp Planning-Based Answering Agent Question WordNet Answer Resolution Answer HitList Search Answer Selection Answer Justification & Presentation Answers Cyc QFrame Web Answering Agents Answering Agent Selection KS Adaptation Layer Answer Classification Question Analysis QPlans QPlan Execution Eng Answer Resolution Answer Candidates Plan Selection QGoals QFilter

10. IBM - PIQUANT IBM Research Subcontractor: Cycorp QPlans Plans for attacking different question types Identifies knowledge sources to use Text Search, Cyc, WordNet, … Specifies preferences, when relevant, of sources Simple questions have base plans (no recursion) Complex questions can be broken into sub-plans

11. IBM - PIQUANT IBM Research Subcontractor: Cycorp Sample Question Types 10 identified, 5 with QPlans When When was the Battle of Hastings? Define What is anorexia nervosa? Property What is the population of the capital of Great Britain? WhatX What county is Phoenix AZ in? Super What is the largest snake in the world?

12. IBM - PIQUANT IBM Research Subcontractor: Cycorp Mapping Questions to QPlans What is the Declaration of Independence? What is the capital of Great Britain? What is the P of X? What is the Declaration of Independence? What is X? What is the capital of Great Britain? What is X? Property Define

13. IBM - PIQUANT IBM Research Subcontractor: Cycorp QPlan Example Ask: “What is the population of the capital of Great Britain?” Recognize question type: Property Recognize answer type: NUMBER/POPULATION Plan Text Search: “Population of the capital of Great Britain” PA Search: “The capital of Great Britain” and (NUMBER$ or POPULATION$) Cyc, DB and WordNet queries Decomposition For each answer, A, to “What is the capital of Great Britain?” Ask: “What is the population of” A Each element of the decomposition may be answered by different knowledge sources (e.g., Cyc, WordNet etc).

14. IBM - PIQUANT IBM Research Subcontractor: Cycorp Our TREC10 System vs. PIQUANT What is the population of the capital of Tajikistan? Text Search 5.3 Million Wrong! What is the capital of Tajikistan? What is the population of Dushanbe? Text Search Cyc X = Dushanbe Cyc What is the population of Dushanbe? 460,000 nil What is the population of the capital of Tajikistan? What is the population of X? Right!

15. IBM - PIQUANT IBM Research Subcontractor: Cycorp PIQUANT Architecture Question WordNet Answer Resolution Answer HitList Search Answer Selection Answer Justification & Presentation Answers Cyc QFrame Web Answering Agents Answering Agent Selection KS Adaptation Layer Answer Classification Question Analysis QPlans QPlan Execution Eng Answer Resolution Answer Candidates Plan Selection QGoals QFilter

16. IBM - PIQUANT IBM Research Subcontractor: Cycorp Enhance Answer Resolution/Selection Deeper linguistic analysis Identifying and matching answer type Name-Entity Tagger Matching syntactic relationships between Q and A Deep Parser Multiple knowledge sources to reinforce answers Encyclopedia Britannica “Crazy Answer” Elimination Using Cyc

17. IBM - PIQUANT IBM Research Subcontractor: Cycorp Deeper Linguistic Analysis In Answer Selection Input Passages (typically 10) returned by the search engine Candidate passages for question: What is the capital of England? “Shaykh Salim Sabah al-Salim continued his talks today with high-ranking officials in the British capital, London.” “BRISTOL, capital of south-west England, holds a peculiar fascination for psephologists.” Semantic type(s) of answer sought Process Identify candidate answers using a semantic-based named-entity tagger Shaykh Salim Sabah al-Salim continued his talks today with high-ranking officials in the British capital, London.” Rank candidate answers based on pre-identified features Hit List (Passages) Answer type Answers & Ranks Answer Selection

18. IBM - PIQUANT IBM Research Subcontractor: Cycorp Multiple Knowledge Sources Question WordNet Answer Resolution Answer Type HitList Text Query Search KB Query Answer Selection Answer Justification & Presentation Answers Text Search Answers Answer Classification Question Analysis Cyc Cyc Answers WordNet Answers EB with PA Index TREC with PA Index Substantiating answers with multiple sources increases confidence TREC Corpus + Encyclopedia Britannica Found previously missed answers Improved rank of previously found answers

19. IBM - PIQUANT IBM Research Subcontractor: Cycorp PIQUANT Architecture Question WordNet Answer Resolution Answer HitList Search Answer Selection Answer Justification & Presentation Answers Cyc QFrame Web Answering Agents Answering Agent Selection KS Adaptation Layer Answer Classification Question Analysis QPlans QPlan Execution Eng Answer Resolution Answer Candidates Plan Selection QGoals QFilter

20. IBM - PIQUANT IBM Research Subcontractor: Cycorp “Crazy Answer” Elimination Semantic type mismatch Examples What city in Florida is Sea World in? London, San Diego, Tulsa Who was Charles Lindbergh’s wife? Babe Ruth, Jack Dempsey Issue Need to determine if an ISA relationship is possible between two entities Unreasonable numerical ranges Examples What is the weight of a wolf? 300 tons How many states have a lottery? 600, 203 How big is our galaxy in diameter? 14 feet, 43 feet Issues (Under Development at Cycorp) Need upper and/or lower bounds on property values Need reasonable units for certain measures

21. IBM - PIQUANT IBM Research Subcontractor: Cycorp Performance Evaluation Evaluation performed on a set of 364 TREC9 questions Results of Improved Answer Selection/Resolution Deeper linguistic analysis Multiple knowledge sources to reinforce answers MRR# Missed Answers # Answers in Rank 1 TREC100.66664203 +Improved Ranking0.72047228 +Multiple Sources0.73942235 +Sanity CheckingTBD Substantially increased number of answers in rank 1 particularly important in recursive architecture

22. IBM - PIQUANT IBM Research Subcontractor: Cycorp Next Six Months Richer question-classification, plan development and execution Ontology synthesis and central management/access Richer and more robust integration of knowledge sources Answer Aggregation Answer Elimination Answer Generation Answering Agent for Causality Questions Leverage dialog with Cyc regarding event pre and post conditions e.g., postCondition (“drink poison”, “die”) Improve Answer Resolution Confidence Processing Implementation Improvements (Speed, Modularity)

23. AQUAINT IBM PIQUANT ARDACycorp Subcontractor: PIQUANT June Workshop Update The End