PIQUANT Question Answering System Dave Ferrucci, John Prager, Jennifer Chu-Carroll, Chris Welty, Chris Cesar and Scott Fahlman ARDA AQUAINT Program June Workshop 2002 This work was supported in part by the Advanced Research and Development Activity (ARDA)'s Advanced Question Answering for Intelligence (AQUAINT) Program under contract number MDA904-01-C-0988.

Subcontractor: Cycorp Overview Progress Update Architecture Qplans Working Example Answer Selection and Resolution Performance Improvements Summary IBM Research Subcontractor: Cycorp

PIQUANT Research Objectives Integration & impact of knowledge based system (e.g., Cyc) in QA Extensible QA architectures Declarative question plans Parallel solution paths and pervasive confidence processing Deeper linguistic & knowledge-based analysis IBM Research Subcontractor: Cycorp

Progress Since AQUAINT Kickoff Architecture Design Support for multiple answering agents, solution paths and knowledge sources Centralized ontology management & uniform access to knowledge sources New question plan modules Improved Ranking Enhanced Answer Selection using deeper linguistic analysis Integration of Cyc in Answer Resolution for “sanity checking” Integration of multiple knowledge sources Answering question previously missed Multiple solutions paths based on alternative question decomposition Integration of Cyc as a knowledge source IBM Research Subcontractor: Cycorp

Architectural Limitations as of TREC10 Pipeline Single Answering Approach Limited Extensibility Single Solution Source WordNet added as second-class citizen No Knowledge System component Limited question understanding Shallow conceptual map from Q to A Limited to explicit matches -- cut-off from inferred possibilities “Explanations” limited to text passages containing answers Can’t filter out crazy answers IBM Research Subcontractor: Cycorp

Classic Pipeline with WordNet Question Analysis Question Search Text Query HitList Answer Classification Answer Type WN Query WN Answer Answer Selection WordNet Answer IBM Research Subcontractor: Cycorp

Knowledge Source Services Question Analysis WordNet Question KB Query Cyc Answer Classification Text Query Search List Hit Answer Type Answer Selection Text Search Answers Cyc Answers Answer Justification & Presentation Answer Resolution Answer Answers WordNet Answers IBM Research Subcontractor: Cycorp

Subcontractor: Cycorp Answering Agents KS Adaptation Layer Answering Agents QGoals WordNet Question Analysis Question Complex Decomposition & Planning QFrame Cyc Answer Classification Causality Search Convert Question to Web Query List Hit Answer Selection Web Answer Justification & Presentation Answer Resolution Answer Answers IBM Research Subcontractor: Cycorp

Planning-Based Answering Agent KS Adaptation Layer Answering Agents WordNet Question Analysis Question Plan Selection QFrame Answer Classification Answering Agent Selection QPlans Search QPlan Execution Eng QGoals List Hit Answer Selection Answer Candidates Web Answer Resolution Cyc QFilter Answer Justification & Presentation Answer Resolution Answer Answers IBM Research Subcontractor: Cycorp

Subcontractor: Cycorp QPlans Plans for attacking different question types Identifies knowledge sources to use Text Search, Cyc, WordNet, … Specifies preferences, when relevant, of sources Simple questions have base plans (no recursion) Complex questions can be broken into sub-plans IBM Research Subcontractor: Cycorp

Sample Question Types 10 identified, 5 with QPlans When When was the Battle of Hastings? Define What is anorexia nervosa? Property What is the population of the capital of Great Britain? WhatX What county is Phoenix AZ in? Super What is the largest snake in the world? IBM Research Subcontractor: Cycorp

Mapping Questions to QPlans Property What is the P of X? What is the P of X? What is the capital of Great Britain? What is the Declaration of Independence? Define What is X? What is X? What is the capital of Great Britain? What is the Declaration of Independence? IBM Research Subcontractor: Cycorp

Subcontractor: Cycorp QPlan Example Ask: “What is the population of the capital of Great Britain?” Recognize question type: Property Recognize answer type: NUMBER/POPULATION Plan Text Search: “Population of the capital of Great Britain” PA Search: “The capital of Great Britain” and (NUMBER$ or POPULATION$) Cyc, DB and WordNet queries Decomposition For each answer, A, to “What is the capital of Great Britain?” Ask: “What is the population of” A Each element of the decomposition may be answered by different knowledge sources (e.g., Cyc, WordNet etc). IBM Research Subcontractor: Cycorp

Our TREC10 System vs. PIQUANT What is the population of the capital of Tajikistan? Text Search Wrong! 5.3 Million What is the capital of Tajikistan? What is the population of Dushanbe? Text Search Cyc X = Dushanbe 460,000 nil What is the population of the capital of Tajikistan? What is the population of X? Right! IBM Research Subcontractor: Cycorp

Subcontractor: Cycorp PIQUANT Architecture KS Adaptation Layer Answering Agents WordNet Question Analysis Question Plan Selection QFrame Answer Classification Answering Agent Selection QPlans Search QPlan Execution Eng QGoals List Hit Answer Selection Answer Candidates Web Answer Resolution Cyc QFilter Answer Justification & Presentation Answer Resolution Answer Answers IBM Research Subcontractor: Cycorp

Enhance Answer Resolution/Selection Deeper linguistic analysis Identifying and matching answer type Name-Entity Tagger Matching syntactic relationships between Q and A Deep Parser Multiple knowledge sources to reinforce answers Encyclopedia Britannica “Crazy Answer” Elimination Using Cyc IBM Research Subcontractor: Cycorp

Deeper Linguistic Analysis In Answer Selection Hit List (Passages) Answer Selection Answers & Ranks Answer type Input Passages (typically 10) returned by the search engine Candidate passages for question: What is the capital of England? “Shaykh Salim Sabah al-Salim continued his talks today with high-ranking officials in the British capital, London.” “BRISTOL, capital of south-west England, holds a peculiar fascination for psephologists.” Semantic type(s) of answer sought Process Identify candidate answers using a semantic-based named-entity tagger <PERSON>Shaykh Salim Sabah al-Salim</PERSON> continued his talks <DATE>today</DATE> with <ROLE>high-ranking officials</ROLE> in the British capital, <CAPITAL>London</CAPITAL>.” Rank candidate answers based on pre-identified features IBM Research Subcontractor: Cycorp

Multiple Knowledge Sources Question Analysis KB Query WordNet Question EB with PA Index Cyc Answer Classification Text Query Search Answer Type List Hit Answer Selection TREC with PA Index Substantiating answers with multiple sources increases confidence TREC Corpus + Encyclopedia Britannica Found previously missed answers Improved rank of previously found answers Text Search Answers Cyc Answers Answer Justification & Presentation Answer Resolution Answer Answers WordNet Answers IBM Research Subcontractor: Cycorp

Subcontractor: Cycorp PIQUANT Architecture KS Adaptation Layer Answering Agents WordNet Question Analysis Question Plan Selection QFrame Answer Classification Answering Agent Selection QPlans Search QPlan Execution Eng QGoals List Hit Answer Selection Answer Candidates Web Answer Resolution Cyc QFilter Answer Justification & Presentation Answer Resolution Answer Answers IBM Research Subcontractor: Cycorp

“Crazy Answer” Elimination Semantic type mismatch Examples What city in Florida is Sea World in? London, San Diego, Tulsa Who was Charles Lindbergh’s wife? Babe Ruth, Jack Dempsey Issue Need to determine if an ISA relationship is possible between two entities Unreasonable numerical ranges What is the weight of a wolf? 300 tons How many states have a lottery? 600, 203 How big is our galaxy in diameter? 14 feet, 43 feet Issues (Under Development at Cycorp) Need upper and/or lower bounds on property values Need reasonable units for certain measures IBM Research Subcontractor: Cycorp

Performance Evaluation Evaluation performed on a set of 364 TREC9 questions Results of Improved Answer Selection/Resolution Deeper linguistic analysis Multiple knowledge sources to reinforce answers MRR # Missed Answers # Answers in Rank 1 TREC10 0.666 64 203 +Improved Ranking 0.720 47 228 +Multiple Sources 0.739 42 235 +Sanity Checking TBD Substantially increased number of answers in rank 1 particularly important in recursive architecture IBM Research Subcontractor: Cycorp

Subcontractor: Cycorp Next Six Months Richer question-classification, plan development and execution Ontology synthesis and central management/access Richer and more robust integration of knowledge sources Answer Aggregation Answer Elimination Answer Generation Answering Agent for Causality Questions Leverage dialog with Cyc regarding event pre and post conditions e.g., postCondition (“drink poison”, “die”) Improve Answer Resolution Confidence Processing Implementation Improvements (Speed, Modularity) IBM Research Subcontractor: Cycorp

PIQUANT June Workshop Update The End