Presentation is loading. Please wait.

Presentation is loading. Please wait.

AQUAINT IBM PIQUANT ARDACYCORP Subcontractor: IBM Question Answering Update piQuAnt ARDA/AQUAINT December 2002 Workshop This work was supported in part.

Published bySara Bradley Modified over 9 years ago

Similar presentations

Presentation on theme: "AQUAINT IBM PIQUANT ARDACYCORP Subcontractor: IBM Question Answering Update piQuAnt ARDA/AQUAINT December 2002 Workshop This work was supported in part."— Presentation transcript:

1 AQUAINT IBM PIQUANT ARDACYCORP Subcontractor: IBM Question Answering Update piQuAnt ARDA/AQUAINT December 2002 Workshop This work was supported in part by the Advanced Research and Development Activity (ARDA)'s Advanced Question Answering for Intelligence (AQUAINT) Program under contract number MDA904-01-C-0988.

2 IBM - PIQUANT IBM Research Subcontractor: Cycorp Overview Progress with QPlans Multi-Agent, Multi-Source Architecture & Answer Resolution 2002 Performance Evaluation

3 IBM - PIQUANT IBM Research Subcontractor: Cycorp Single Strategy -> Plan-Based More sophisticated Question Analysis using full parse and NE recognition Search strategy based on type of question. Any or all of following: Regular – Predictive Annotation Relative – Relative clauses, appositions Definition – Use external structured knowledge (WordNet, Tables from WWW, Databases, Cyc) Corpus strategy (selected by user) Answering Agent strategy (selected by user)

4 IBM - PIQUANT IBM Research Subcontractor: Cycorp A Single-Agent, Single-Strategy QA Architecture Question Answer Presentation Answers NLP Utilities Answer Classification Question Analysis Search HitList WordNet Answer Selection SE Query Answer Type

5 IBM - PIQUANT IBM Research Subcontractor: Cycorp A Multi-Agent QA Architecture KSP-Based Answering Agent Rule-Based Answering Agent Answering Agents Statistical Answering Agent Definitional Q Answering Agent Question Answer Resolution Answer QGoals Answer Justification & Presentation Answers QFrame NLP Utilities Answer Classification Question Analysis Search HitList WordNet Cyc Answer Selection Web KS Adaptation Layer Web-Based Answering Agent QPlan Generator QPlan Executor

6 IBM - PIQUANT IBM Research Subcontractor: Cycorp A Multi-Agent QA Architecture KSP-Based Answering Agent Rule-Based Answering Agent Answering Agents Statistical Answering Agent Definitional Q Answering Agent Question Answer Resolution Answer QGoals Answer Justification & Presentation Answers QFrame NLP Utilities Answer Classification Question Analysis Search HitList WordNet Cyc Answer Selection Web KS Adaptation Layer Web-Based Answering Agent QPlan Generator QPlan Executor

7 AQUAINT IBM PIQUANT ARDACYCORP Subcontractor: Merging and Resolving Answers in a Multi-Agent QA Architecture Jennifer Chu-Carroll November 8, 2002

8 IBM - PIQUANT IBM Research Subcontractor: Cycorp A Multi-Agent QA Architecture KSP-Based Answering Agent Rule-Based Answering Agent Answering Agents Statistical Answering Agent Definitional Q Answering Agent Question Answer Resolution Answer QGoals Answer Justification & Presentation Answers QFrame NLP Utilities Answer Classification Question Analysis Search HitList WordNet Cyc Answer Selection Web KS Adaptation Layer (OntASK) Web-Based Answering Agent TREC 10 TREC 11 EB

9 IBM - PIQUANT IBM Research Subcontractor: Cycorp Currently Implemented Answering Agents Agents based on unstructured information Agent strategies: Knowledge-based answering agent Statistical answering agent Knowledge sources: AQUAINT corpus TREC corpus Encyclopedia Britannica Agents based on structured information Agent strategies: Knowledge source query via KSP Sanity checking: Post-hoc filtering of candidate answers Knowledge sources WordNet Cyc Databases

10 IBM - PIQUANT IBM Research Subcontractor: Cycorp Answer Resolution Combine answers from multiple answering agents Question Analysis 1 Search 1 Corpus 1 Answer Selection 1 Question Analysis 2 Search 2 Answer Selection 2 Confidence Reranking Answer Resolution passages answers Question Analysis 1 Search 1 Corpus 2 Answer Selection 1 Agent 1 Agent 2 Agent 3 Sanity Checking Final answer

11 IBM - PIQUANT IBM Research Subcontractor: Cycorp Answer Resolution Components Answer Selection Combines answers proposed by passages retrieved Using different keywords and/or search strategies From different corpora using the same strategy Motivation Different strategies/corpora may produce different relevant passages Semantically-equivalent answers appearing in different contexts Enables answer selection to better find close matches with question Reinforces one another A corpus may be Primary corpus: answers can be proposed and justified Supporting corpus: answers can only support those found in the primary corpus

12 IBM - PIQUANT IBM Research Subcontractor: Cycorp Answer Selection Process Identifies candidate answers and their semantic types Evaluates candidate answers based on Semantic type match Grammatical relationship match Performs candidate answer normalization E.g., Clinton = Bill Clinton = President Clinton Currently focuses on named entity normalization Combines evidence for each candidate answer and computes score

13 IBM - PIQUANT IBM Research Subcontractor: Cycorp Answer Selection Example TREC11 Q: “How many chromosomes does a human zygote have?” Passages from rule-based strategy + AQUAINT corpus Of the 46 human chromosomes, 44 are identical pairs. There are 46 paired chromosomes in a human being’s cell nucleus. … the order of the 21 st of the 23 pairs of human chromosome, … … narrowed their search of the gene to a small section of human chromosome 7, … … fused together to form the present-day human chromosome 7. System returns “7” as its top answer

14 IBM - PIQUANT IBM Research Subcontractor: Cycorp Answer Selection Example (Cont’d) TREC11 Q: “How many chromosomes does a human zygote have?” Passages from statistical strategy + AQUAINT corpus … sequence the roughly 100,000 genes on the 46 human chromosomes. Of the 46 human chromosomes, 44 are identical pairs. Passages from rule-based strategy + TREC corpus There are 46 chromosomes in a normal human cell. … located on one of the 46 chromosomes in every human cell. Passages from rule-based strategy + Encyclopedia Britannica In each body cell of normal human beings, there are 46 chromosomes, … Normally, humans have 46 chromosomes arranged in 23 pairs. With additional passages, system now returns “46” as its top answer

15 IBM - PIQUANT IBM Research Subcontractor: Cycorp Answer Resolution Components Confidence Reranking Invoked only if two or more strategically-independent answering agent are used Motivation Better confidence in the same answer given by two strategically- independent agents Process Adjust confidence scores of previously-determined answers in consultation with another answer set Score receives large boost if identical answer given by other answer agent Score receives small boost if partially overlapping answer given

16 IBM - PIQUANT IBM Research Subcontractor: Cycorp Answer Resolution Components Cyc Sanity Checker A post-hoc process for Rejecting “insane” answers How much does a grey wolf weigh? 300 tons Boosting confidence for “sane” answers Sanity checker invoked with Predicate, e.g. “weight” Focus, e.g. “grey wolf” Candidate value, e.g. “300 tons” Sanity checker returns “Sane”: + or – 10% of value in Cyc “Insane”: outside of the reasonable range “Don’t know” Confidence score highly boosted when answer is “sane”

17 IBM - PIQUANT IBM Research Subcontractor: Cycorp Cyc Sanity Checking Example Trec11 Q: “What is the population of Maryland?” Without sanity checking PIQUANT’s top answer: “50,000” Justification: “Maryland’s population is 50,000 and growing rapidly.” Passage discusses an exotic species “nutria”, not humans With sanity checking Cyc knows the population of Maryland is 5,296,486 It rejects the top “insane” answers PIQUANT’s new top answer: “5.1 million” with very high confidence

18 IBM - PIQUANT IBM Research Subcontractor: Cycorp Performance Evaluation Conducted experiments to evaluate the multi-source and multi- strategy aspects of PIQUANT System configurations TREC2001 system: pre-AQUAINT Single Source & Single Strategy: Strategy: Rule-based or statistical answering agent Source: AQUAINT corpus Multiple Sources & Single Strategy: Strategy: Rule-based answering agent Sources Primary: AQUAINT corpus Supporting: TREC corpus, EB Multiple Sources & Multiple Strategies: Strategies: Rule-based and statistical answering agents Sources Primary: AQUAINT corpus Supporting: TREC corpus, EB

19 IBM - PIQUANT IBM Research Subcontractor: Cycorp Evaluation Results Overall impact of agents based on unstructured information 41.3% relative improvement in number of questions correctly answered 51.8% relative improvement in average precision Impact of agents based on structured knowledge sources KSP invoked 5 times, returned 5 correct answers Cyc sanity checker invoked 3 times, returned 1 definitive answer TREC2001 system Single source Rule-based strategy Single source Statistical strategy Multiple sources Rule-based strategy Multiple sources Multiple strategies % correct28.3%32.5%32.7%38.2%40.0% Avg prec0.4460.5310.5340.6220.677

Download ppt "AQUAINT IBM PIQUANT ARDACYCORP Subcontractor: IBM Question Answering Update piQuAnt ARDA/AQUAINT December 2002 Workshop This work was supported in part."

Similar presentations

The 20th International Conference on Software Engineering and Knowledge Engineering (SEKE2008) Department of Electrical and Computer Engineering

The 20th International Conference on Software Engineering and Knowledge Engineering (SEKE2008) Department of Electrical and Computer Engineering
SEARCHING QUESTION AND ANSWER ARCHIVES Dr. Jiwoon Jeon Presented by CHARANYA VENKATESH KUMAR.

SEARCHING QUESTION AND ANSWER ARCHIVES Dr. Jiwoon Jeon Presented by CHARANYA VENKATESH KUMAR.
Overview of the KBP 2013 Slot Filler Validation Track Hoa Trang Dang National Institute of Standards and Technology.

Overview of the KBP 2013 Slot Filler Validation Track Hoa Trang Dang National Institute of Standards and Technology.
10-803 Markov Logic Networks Instructor: Pedro Domingos.

Markov Logic Networks Instructor: Pedro Domingos.
Chapter 2. Slide 1 CULTURAL SUBJECT GATEWAYS CULTURAL SUBJECT GATEWAYS Subject Gateways  Started as links of lists  Continued as Web directories  Culminated.

Chapter 2. Slide 1 CULTURAL SUBJECT GATEWAYS CULTURAL SUBJECT GATEWAYS Subject Gateways  Started as links of lists  Continued as Web directories  Culminated.
Vikas BhardwajColumbia University NLP for the Web – Spring 2010 Improving QA Accuracy by Question Inversion Prager et al. IBM T.J. Watson Res. Ctr. 02/18/2010.

Vikas BhardwajColumbia University NLP for the Web – Spring 2010 Improving QA Accuracy by Question Inversion Prager et al. IBM T.J. Watson Res. Ctr. 02/18/2010.
Question-Answering: Overview Ling573 Systems & Applications March 31, 2011.

Question-Answering: Overview Ling573 Systems & Applications March 31, 2011.
Evaluation.  Allan, Ballesteros, Croft, and/or Turtle Types of Evaluation Might evaluate several aspects Evaluation generally comparative –System A vs.

Evaluation.  Allan, Ballesteros, Croft, and/or Turtle Types of Evaluation Might evaluate several aspects Evaluation generally comparative –System A vs.
NaLIX: A Generic Natural Language Search Environment for XML Data Presented by: Erik Mathisen 02/12/2008.

NaLIX: A Generic Natural Language Search Environment for XML Data Presented by: Erik Mathisen 02/12/2008.
CSE 574: Artificial Intelligence II Statistical Relational Learning Instructor: Pedro Domingos.

CSE 574: Artificial Intelligence II Statistical Relational Learning Instructor: Pedro Domingos.
Using Information Extraction for Question Answering Done by Rani Qumsiyeh.

Using Information Extraction for Question Answering Done by Rani Qumsiyeh.
21 21 Web Content Management Architectures Vagan Terziyan MIT Department, University of Jyvaskyla, AI Department, Kharkov National University of Radioelectronics.

21 21 Web Content Management Architectures Vagan Terziyan MIT Department, University of Jyvaskyla, AI Department, Kharkov National University of Radioelectronics.
Information retrieval Finding relevant data using irrelevant keys Example: database of photographic images sorted by number, date. DBMS: Well structured.

Information retrieval Finding relevant data using irrelevant keys Example: database of photographic images sorted by number, date. DBMS: Well structured.
An investigation of query expansion terms Gheorghe Muresan Rutgers University, School of Communication, Information and Library Science 4 Huntington St.,

An investigation of query expansion terms Gheorghe Muresan Rutgers University, School of Communication, Information and Library Science 4 Huntington St.,
University of Kansas Data Discovery on the Information Highway Susan Gauch University of Kansas.

University of Kansas Data Discovery on the Information Highway Susan Gauch University of Kansas.
Enhance legal retrieval applications with an automatically induced knowledge base Ka Kan Lo.

Enhance legal retrieval applications with an automatically induced knowledge base Ka Kan Lo.
A Web-based Question Answering System Yu-shan & Wenxiu 03.08.2005.

A Web-based Question Answering System Yu-shan & Wenxiu
AQUAINT Kickoff Meeting – December 2001 Integrating Robust Semantics, Event Detection, Information Fusion, and Summarization for Multimedia Question Answering.

AQUAINT Kickoff Meeting – December 2001 Integrating Robust Semantics, Event Detection, Information Fusion, and Summarization for Multimedia Question Answering.
Faculty of Informatics and Information Technologies Slovak University of Technology Personalized Navigation in the Semantic Web Michal Tvarožek Mentor:

Faculty of Informatics and Information Technologies Slovak University of Technology Personalized Navigation in the Semantic Web Michal Tvarožek Mentor:
JAVELIN Project Briefing 1 AQUAINT Year I Mid-Year Review Language Technologies Institute Carnegie Mellon University Status Update for Mid-Year Program.

JAVELIN Project Briefing 1 AQUAINT Year I Mid-Year Review Language Technologies Institute Carnegie Mellon University Status Update for Mid-Year Program.

Similar presentations

Ads by Google