1. PIQUANT Question Answering System
Dave Ferrucci, John Prager, Jennifer Chu-Carroll, Chris Welty, Chris Cesar and Scott Fahlman
ARDA AQUAINT Program
June Workshop 2002
This work was supported in part by the Advanced Research and Development Activity (ARDA)'s Advanced Question Answering for Intelligence (AQUAINT) Program under contract number MDA C-0988.

2. Subcontractor: Cycorp
Overview
Progress Update
Architecture
Qplans
Working Example
Answer Selection and Resolution
Performance Improvements
Summary
IBM Research
Subcontractor: Cycorp

3. PIQUANT Research Objectives
Integration & impact of knowledge based system (e.g., Cyc) in QA
Extensible QA architectures
Declarative question plans
Parallel solution paths and pervasive confidence processing
Deeper linguistic & knowledge-based analysis
IBM Research
Subcontractor: Cycorp

4. Progress Since AQUAINT Kickoff
Architecture Design
Support for multiple answering agents, solution paths and knowledge sources
Centralized ontology management & uniform access to knowledge sources
New question plan modules
Improved Ranking
Enhanced Answer Selection using deeper linguistic analysis
Integration of Cyc in Answer Resolution for “sanity checking”
Integration of multiple knowledge sources
Answering question previously missed
Multiple solutions paths based on alternative question decomposition
Integration of Cyc as a knowledge source
IBM Research
Subcontractor: Cycorp

5. Architectural Limitations as of TREC10
Pipeline
Single Answering Approach
Limited Extensibility
Single Solution Source
WordNet added as second-class citizen
No Knowledge System component
Limited question understanding
Shallow conceptual map from Q to A
Limited to explicit matches -- cut-off from inferred possibilities
“Explanations” limited to text passages containing answers
Can’t filter out crazy answers
IBM Research
Subcontractor: Cycorp

6. Classic Pipeline with WordNet
Question
Analysis
Question
Search
Text Query
HitList
Answer
Classification
Answer Type
WN Query
WN Answer
Answer
Selection
WordNet
Answer
IBM Research
Subcontractor: Cycorp

7. Knowledge Source Services
Question
Analysis
WordNet
Question
KB Query
Cyc
Answer
Classification
Text
Query
Search
List
Hit
Answer
Type
Answer
Selection
Text Search Answers
Cyc Answers
Answer
Justification
& Presentation
Answer
Resolution
Answer
Answers
WordNet Answers
IBM Research
Subcontractor: Cycorp

8. Subcontractor: Cycorp
Answering Agents
KS Adaptation Layer
Answering Agents
QGoals
WordNet
Question
Analysis
Question
Complex
Decomposition
& Planning
QFrame
Cyc
Answer
Classification
Causality
Search
Convert
Question to
Web Query
List
Hit
Answer
Selection
Web
Answer
Justification
& Presentation
Answer
Resolution
Answer
Answers
IBM Research
Subcontractor: Cycorp

9. Planning-Based Answering Agent
KS Adaptation Layer
Answering Agents
WordNet
Question
Analysis
Question
Plan
Selection
QFrame
Answer
Classification
Answering
Agent
Selection
QPlans
Search
QPlan Execution Eng
QGoals
List
Hit
Answer
Selection
Answer
Candidates
Web
Answer
Resolution
Cyc
QFilter
Answer
Justification
& Presentation
Answer
Resolution
Answer
Answers
IBM Research
Subcontractor: Cycorp

10. Subcontractor: Cycorp
QPlans
Plans for attacking different question types
Identifies knowledge sources to use
Text Search, Cyc, WordNet, …
Specifies preferences, when relevant, of sources
Simple questions have base plans (no recursion)
Complex questions can be broken into sub-plans
IBM Research
Subcontractor: Cycorp

11. Sample Question Types 10 identified, 5 with QPlans
When
When was the Battle of Hastings?
Define
What is anorexia nervosa?
Property
What is the population of the capital of Great Britain?
WhatX
What county is Phoenix AZ in?
Super
What is the largest snake in the world?
IBM Research
Subcontractor: Cycorp

12. Mapping Questions to QPlans
Property
What is the P of X?
What is the P of X?
What is the capital of Great Britain?
What is the Declaration of Independence?
Define
What is X?
What is X?
What is the capital of Great Britain?
What is the Declaration of Independence?
IBM Research
Subcontractor: Cycorp

13. Subcontractor: Cycorp
QPlan Example
Ask: “What is the population of the capital of Great Britain?”
Recognize question type: Property
Recognize answer type: NUMBER/POPULATION
Plan
Text Search: “Population of the capital of Great Britain”
PA Search: “The capital of Great Britain” and (NUMBER$ or POPULATION$)
Cyc, DB and WordNet queries
Decomposition
For each answer, A, to “What is the capital of Great Britain?”
Ask: “What is the population of” A
Each element of the decomposition may be answered by different knowledge sources (e.g., Cyc, WordNet etc).
IBM Research
Subcontractor: Cycorp

14. Our TREC10 System vs. PIQUANT
What is the population of the capital of Tajikistan?
Text
Search
Wrong!
5.3 Million
What is the capital of Tajikistan?
What is the population of Dushanbe?
Text
Search
Cyc
X = Dushanbe
460,000
nil
What is the population of the capital of Tajikistan?
What is the population of X?
Right!
IBM Research
Subcontractor: Cycorp

15. Subcontractor: Cycorp
PIQUANT Architecture
KS Adaptation Layer
Answering Agents
WordNet
Question
Analysis
Question
Plan
Selection
QFrame
Answer
Classification
Answering
Agent
Selection
QPlans
Search
QPlan Execution Eng
QGoals
List
Hit
Answer
Selection
Answer
Candidates
Web
Answer
Resolution
Cyc
QFilter
Answer
Justification
& Presentation
Answer
Resolution
Answer
Answers
IBM Research
Subcontractor: Cycorp

16. Enhance Answer Resolution/Selection
Deeper linguistic analysis
Identifying and matching answer type
Name-Entity Tagger
Matching syntactic relationships between Q and A
Deep Parser
Multiple knowledge sources to reinforce answers
Encyclopedia Britannica
“Crazy Answer” Elimination
Using Cyc
IBM Research
Subcontractor: Cycorp

17. Deeper Linguistic Analysis In Answer Selection
Hit List (Passages)
Answer
Selection
Answers & Ranks
Answer type
Input
Passages (typically 10) returned by the search engine
Candidate passages for question: What is the capital of England?
“Shaykh Salim Sabah al-Salim continued his talks today with high-ranking officials in the British capital, London.”
“BRISTOL, capital of south-west England, holds a peculiar fascination for psephologists.”
Semantic type(s) of answer sought
Process
Identify candidate answers using a semantic-based named-entity tagger
<PERSON>Shaykh Salim Sabah al-Salim</PERSON> continued his talks <DATE>today</DATE> with <ROLE>high-ranking officials</ROLE> in the British capital, <CAPITAL>London</CAPITAL>.”
Rank candidate answers based on pre-identified features
IBM Research
Subcontractor: Cycorp

18. Multiple Knowledge Sources
Question
Analysis
KB Query
WordNet
Question EB
with
PA Index
Cyc
Answer
Classification
Text
Query
Search
Answer
Type
List
Hit
Answer
Selection
TREC
with
PA Index
Substantiating answers with multiple sources increases confidence
TREC Corpus + Encyclopedia Britannica
Found previously missed answers
Improved rank of previously found answers
Text Search Answers
Cyc Answers
Answer
Justification
& Presentation
Answer
Resolution
Answer
Answers
WordNet Answers
IBM Research
Subcontractor: Cycorp

19. Subcontractor: Cycorp
PIQUANT Architecture
KS Adaptation Layer
Answering Agents
WordNet
Question
Analysis
Question
Plan
Selection
QFrame
Answer
Classification
Answering
Agent
Selection
QPlans
Search
QPlan Execution Eng
QGoals
List
Hit
Answer
Selection
Answer
Candidates
Web
Answer
Resolution
Cyc
QFilter
Answer
Justification
& Presentation
Answer
Resolution
Answer
Answers
IBM Research
Subcontractor: Cycorp

20. “Crazy Answer” Elimination
Semantic type mismatch
Examples
What city in Florida is Sea World in?
London, San Diego, Tulsa
Who was Charles Lindbergh’s wife?
Babe Ruth, Jack Dempsey
Issue
Need to determine if an ISA relationship is possible between two entities
Unreasonable numerical ranges
What is the weight of a wolf?
300 tons
How many states have a lottery?
600, 203
How big is our galaxy in diameter?
14 feet, 43 feet
Issues (Under Development at Cycorp)
Need upper and/or lower bounds on property values
Need reasonable units for certain measures
IBM Research
Subcontractor: Cycorp

21. Performance Evaluation
Evaluation performed on a set of 364 TREC9 questions
Results of Improved Answer Selection/Resolution
Deeper linguistic analysis
Multiple knowledge sources to reinforce answers
MRR
# Missed
Answers
# Answers
in Rank 1
TREC10
0.666 64
203
+Improved Ranking
0.720 47
228
+Multiple Sources
0.739 42
235
+Sanity Checking
TBD
Substantially increased number of answers in rank 1 particularly important in recursive architecture
IBM Research
Subcontractor: Cycorp

22. Subcontractor: Cycorp
Next Six Months
Richer question-classification, plan development and execution
Ontology synthesis and central management/access
Richer and more robust integration of knowledge sources
Answer Aggregation
Answer Elimination
Answer Generation
Answering Agent for Causality Questions
Leverage dialog with Cyc regarding event pre and post conditions
e.g., postCondition (“drink poison”, “die”)
Improve Answer Resolution
Confidence Processing
Implementation Improvements (Speed, Modularity)
IBM Research
Subcontractor: Cycorp

23. PIQUANT June Workshop Update
The End