AQUAINT IBM PIQUANT ARDACYCORP Subcontractor: IBM Question Answering Update piQuAnt ARDA/AQUAINT December 2002 Workshop This work was supported in part by the Advanced Research and Development Activity (ARDA)'s Advanced Question Answering for Intelligence (AQUAINT) Program under contract number MDA C-0988.

IBM - PIQUANT IBM Research Subcontractor: Cycorp Overview Progress with QPlans Multi-Agent, Multi-Source Architecture & Answer Resolution 2002 Performance Evaluation

IBM - PIQUANT IBM Research Subcontractor: Cycorp Single Strategy -> Plan-Based More sophisticated Question Analysis using full parse and NE recognition Search strategy based on type of question. Any or all of following: Regular – Predictive Annotation Relative – Relative clauses, appositions Definition – Use external structured knowledge (WordNet, Tables from WWW, Databases, Cyc) Corpus strategy (selected by user) Answering Agent strategy (selected by user)

IBM - PIQUANT IBM Research Subcontractor: Cycorp A Single-Agent, Single-Strategy QA Architecture Question Answer Presentation Answers NLP Utilities Answer Classification Question Analysis Search HitList WordNet Answer Selection SE Query Answer Type

IBM - PIQUANT IBM Research Subcontractor: Cycorp A Multi-Agent QA Architecture KSP-Based Answering Agent Rule-Based Answering Agent Answering Agents Statistical Answering Agent Definitional Q Answering Agent Question Answer Resolution Answer QGoals Answer Justification & Presentation Answers QFrame NLP Utilities Answer Classification Question Analysis Search HitList WordNet Cyc Answer Selection Web KS Adaptation Layer Web-Based Answering Agent QPlan Generator QPlan Executor

IBM - PIQUANT IBM Research Subcontractor: Cycorp A Multi-Agent QA Architecture KSP-Based Answering Agent Rule-Based Answering Agent Answering Agents Statistical Answering Agent Definitional Q Answering Agent Question Answer Resolution Answer QGoals Answer Justification & Presentation Answers QFrame NLP Utilities Answer Classification Question Analysis Search HitList WordNet Cyc Answer Selection Web KS Adaptation Layer Web-Based Answering Agent QPlan Generator QPlan Executor

AQUAINT IBM PIQUANT ARDACYCORP Subcontractor: Merging and Resolving Answers in a Multi-Agent QA Architecture Jennifer Chu-Carroll November 8, 2002

IBM - PIQUANT IBM Research Subcontractor: Cycorp A Multi-Agent QA Architecture KSP-Based Answering Agent Rule-Based Answering Agent Answering Agents Statistical Answering Agent Definitional Q Answering Agent Question Answer Resolution Answer QGoals Answer Justification & Presentation Answers QFrame NLP Utilities Answer Classification Question Analysis Search HitList WordNet Cyc Answer Selection Web KS Adaptation Layer (OntASK) Web-Based Answering Agent TREC 10 TREC 11 EB

IBM - PIQUANT IBM Research Subcontractor: Cycorp Currently Implemented Answering Agents Agents based on unstructured information Agent strategies: Knowledge-based answering agent Statistical answering agent Knowledge sources: AQUAINT corpus TREC corpus Encyclopedia Britannica Agents based on structured information Agent strategies: Knowledge source query via KSP Sanity checking: Post-hoc filtering of candidate answers Knowledge sources WordNet Cyc Databases

IBM - PIQUANT IBM Research Subcontractor: Cycorp Answer Resolution Combine answers from multiple answering agents Question Analysis 1 Search 1 Corpus 1 Answer Selection 1 Question Analysis 2 Search 2 Answer Selection 2 Confidence Reranking Answer Resolution passages answers Question Analysis 1 Search 1 Corpus 2 Answer Selection 1 Agent 1 Agent 2 Agent 3 Sanity Checking Final answer

IBM - PIQUANT IBM Research Subcontractor: Cycorp Answer Resolution Components Answer Selection Combines answers proposed by passages retrieved Using different keywords and/or search strategies From different corpora using the same strategy Motivation Different strategies/corpora may produce different relevant passages Semantically-equivalent answers appearing in different contexts Enables answer selection to better find close matches with question Reinforces one another A corpus may be Primary corpus: answers can be proposed and justified Supporting corpus: answers can only support those found in the primary corpus

IBM - PIQUANT IBM Research Subcontractor: Cycorp Answer Selection Process Identifies candidate answers and their semantic types Evaluates candidate answers based on Semantic type match Grammatical relationship match Performs candidate answer normalization E.g., Clinton = Bill Clinton = President Clinton Currently focuses on named entity normalization Combines evidence for each candidate answer and computes score

IBM - PIQUANT IBM Research Subcontractor: Cycorp Answer Selection Example TREC11 Q: “How many chromosomes does a human zygote have?” Passages from rule-based strategy + AQUAINT corpus Of the 46 human chromosomes, 44 are identical pairs. There are 46 paired chromosomes in a human being’s cell nucleus. … the order of the 21 st of the 23 pairs of human chromosome, … … narrowed their search of the gene to a small section of human chromosome 7, … … fused together to form the present-day human chromosome 7. System returns “7” as its top answer

IBM - PIQUANT IBM Research Subcontractor: Cycorp Answer Selection Example (Cont’d) TREC11 Q: “How many chromosomes does a human zygote have?” Passages from statistical strategy + AQUAINT corpus … sequence the roughly 100,000 genes on the 46 human chromosomes. Of the 46 human chromosomes, 44 are identical pairs. Passages from rule-based strategy + TREC corpus There are 46 chromosomes in a normal human cell. … located on one of the 46 chromosomes in every human cell. Passages from rule-based strategy + Encyclopedia Britannica In each body cell of normal human beings, there are 46 chromosomes, … Normally, humans have 46 chromosomes arranged in 23 pairs. With additional passages, system now returns “46” as its top answer

IBM - PIQUANT IBM Research Subcontractor: Cycorp Answer Resolution Components Confidence Reranking Invoked only if two or more strategically-independent answering agent are used Motivation Better confidence in the same answer given by two strategically- independent agents Process Adjust confidence scores of previously-determined answers in consultation with another answer set Score receives large boost if identical answer given by other answer agent Score receives small boost if partially overlapping answer given

IBM - PIQUANT IBM Research Subcontractor: Cycorp Answer Resolution Components Cyc Sanity Checker A post-hoc process for Rejecting “insane” answers How much does a grey wolf weigh? 300 tons Boosting confidence for “sane” answers Sanity checker invoked with Predicate, e.g. “weight” Focus, e.g. “grey wolf” Candidate value, e.g. “300 tons” Sanity checker returns “Sane”: + or – 10% of value in Cyc “Insane”: outside of the reasonable range “Don’t know” Confidence score highly boosted when answer is “sane”

IBM - PIQUANT IBM Research Subcontractor: Cycorp Cyc Sanity Checking Example Trec11 Q: “What is the population of Maryland?” Without sanity checking PIQUANT’s top answer: “50,000” Justification: “Maryland’s population is 50,000 and growing rapidly.” Passage discusses an exotic species “nutria”, not humans With sanity checking Cyc knows the population of Maryland is 5,296,486 It rejects the top “insane” answers PIQUANT’s new top answer: “5.1 million” with very high confidence

IBM - PIQUANT IBM Research Subcontractor: Cycorp Performance Evaluation Conducted experiments to evaluate the multi-source and multi- strategy aspects of PIQUANT System configurations TREC2001 system: pre-AQUAINT Single Source & Single Strategy: Strategy: Rule-based or statistical answering agent Source: AQUAINT corpus Multiple Sources & Single Strategy: Strategy: Rule-based answering agent Sources Primary: AQUAINT corpus Supporting: TREC corpus, EB Multiple Sources & Multiple Strategies: Strategies: Rule-based and statistical answering agents Sources Primary: AQUAINT corpus Supporting: TREC corpus, EB

IBM - PIQUANT IBM Research Subcontractor: Cycorp Evaluation Results Overall impact of agents based on unstructured information 41.3% relative improvement in number of questions correctly answered 51.8% relative improvement in average precision Impact of agents based on structured knowledge sources KSP invoked 5 times, returned 5 correct answers Cyc sanity checker invoked 3 times, returned 1 definitive answer TREC2001 system Single source Rule-based strategy Single source Statistical strategy Multiple sources Rule-based strategy Multiple sources Multiple strategies % correct28.3%32.5%32.7%38.2%40.0% Avg prec