Passage Retrieval & Re-ranking Ling573 NLP Systems and Applications May 5, 2011.

Slides:



Advertisements
Similar presentations
Pseudo-Relevance Feedback For Multimedia Retrieval By Rong Yan, Alexander G. and Rong Jin Mwangi S. Kariuki
Advertisements

Diversified Retrieval as Structured Prediction Redundancy, Diversity, and Interdependent Document Relevance (IDR ’09) SIGIR 2009 Workshop Yisong Yue Cornell.
Elliot Holt Kelly Peterson. D4 – Smells Like D3 Primary Goal – improve D3 MAP with lessons learned After many experiments: TREC 2004 MAP = >
QA-LaSIE Components The question document and each candidate answer document pass through all nine components of the QA-LaSIE system in the order shown.
SEARCHING QUESTION AND ANSWER ARCHIVES Dr. Jiwoon Jeon Presented by CHARANYA VENKATESH KUMAR.
Evaluating Search Engine
Answer Extraction Ling573 NLP Systems and Applications May 19, 2011.
Passage Retrieval and Re-ranking Ling573 NLP Systems and Applications May 3, 2011.
A Basic Q/A System: Passage Retrieval. Outline  Query Expansion  Document Ranking  Passage Retrieval  Passage Re-ranking.
Information Retrieval Ling573 NLP Systems and Applications April 26, 2011.
Question-Answering: Overview Ling573 Systems & Applications March 31, 2011.
1 Quasi-Synchronous Grammars  Based on key observations in MT: translated sentences often have some isomorphic syntactic structure, but not usually in.
Re-ranking Documents Segments To Improve Access To Relevant Content in Information Retrieval Gary Madden Applied Computational Linguistics Dublin City.
1 QA in Discussion Boards  Companies (e.g., Dell, IBM) use discussion boards as ways for customers to get answers to their questions  90% of 40 analyzed.
Vector Space Model CS 652 Information Extraction and Integration.
Latent Semantic Analysis (LSA). Introduction to LSA Learning Model Uses Singular Value Decomposition (SVD) to simulate human learning of word and passage.
Employing Two Question Answering Systems in TREC 2005 Harabagiu, Moldovan, et al 2005 Language Computer Corporation.
Finding Advertising Keywords on Web Pages Scott Wen-tau YihJoshua Goodman Microsoft Research Vitor R. Carvalho Carnegie Mellon University.
Information Retrieval in Practice
Tree Kernels for Parsing: (Collins & Duffy, 2001) Advanced Statistical Methods in NLP Ling 572 February 28, 2012.
Hang Cui et al. NUS at TREC-13 QA Main Task 1/20 National University of Singapore at the TREC- 13 Question Answering Main Task Hang Cui Keya Li Renxu Sun.
Query Processing: Query Formulation Ling573 NLP Systems and Applications April 14, 2011.
Combining Lexical Semantic Resources with Question & Answer Archives for Translation-Based Answer Finding Delphine Bernhard and Iryna Gurevvch Ubiquitous.
COMPUTER-ASSISTED PLAGIARISM DETECTION PRESENTER: CSCI 6530 STUDENT.
Thanks to Bill Arms, Marti Hearst Documents. Last time Size of information –Continues to grow IR an old field, goes back to the ‘40s IR iterative process.
Natural Language Based Reformulation Resource and Web Exploitation for Question Answering Ulf Hermjakob, Abdessamad Echihabi, Daniel Marcu University of.
Latent Semantic Analysis Hongning Wang Recap: vector space model Represent both doc and query by concept vectors – Each concept defines one dimension.
21/11/2002 The Integration of Lexical Knowledge and External Resources for QA Hui YANG, Tat-Seng Chua Pris, School of Computing.
Date: 2014/02/25 Author: Aliaksei Severyn, Massimo Nicosia, Aleessandro Moschitti Source: CIKM’13 Advisor: Jia-ling Koh Speaker: Chen-Yu Huang Building.
Generic text summarization using relevance measure and latent semantic analysis Gong Yihong and Xin Liu SIGIR, April 2015 Yubin Lim.
INTERESTING NUGGETS AND THEIR IMPACT ON DEFINITIONAL QUESTION ANSWERING Kian-Wei Kor, Tat-Seng Chua Department of Computer Science School of Computing.
Predicting Question Quality Bruce Croft and Stephen Cronen-Townsend University of Massachusetts Amherst.
Collocations and Information Management Applications Gregor Erbach Saarland University Saarbrücken.
LANGUAGE MODELS FOR RELEVANCE FEEDBACK Lee Won Hee.
CLEF2003 Forum/ August 2003 / Trondheim / page 1 Report on CLEF-2003 ML4 experiments Extracting multilingual resources from corpora N. Cancedda, H. Dejean,
Chapter 23: Probabilistic Language Models April 13, 2004.
Information Retrieval at NLC Jianfeng Gao NLC Group, Microsoft Research China.
Chapter 8 Evaluating Search Engine. Evaluation n Evaluation is key to building effective and efficient search engines  Measurement usually carried out.
Ling573 NLP Systems and Applications May 7, 2013.
Improving Named Entity Translation Combining Phonetic and Semantic Similarities Fei Huang, Stephan Vogel, Alex Waibel Language Technologies Institute School.
A Word Clustering Approach for Language Model-based Sentence Retrieval in Question Answering Systems Saeedeh Momtazi, Dietrich Klakow University of Saarland,Germany.
August 17, 2005Question Answering Passage Retrieval Using Dependency Parsing 1/28 Question Answering Passage Retrieval Using Dependency Parsing Hang Cui.
Supertagging CMSC Natural Language Processing January 31, 2006.
Evaluating Answer Validation in multi- stream Question Answering Álvaro Rodrigo, Anselmo Peñas, Felisa Verdejo UNED NLP & IR group nlp.uned.es The Second.
UWMS Data Mining Workshop Content Analysis: Automated Summarizing Prof. Marti Hearst SIMS 202, Lecture 16.
Comparing Document Segmentation for Passage Retrieval in Question Answering Jorg Tiedemann University of Groningen presented by: Moy’awiah Al-Shannaq
Mining Dependency Relations for Query Expansion in Passage Retrieval Renxu Sun, Chai-Huat Ong, Tat-Seng Chua National University of Singapore SIGIR2006.
Answer Mining by Combining Extraction Techniques with Abductive Reasoning Sanda Harabagiu, Dan Moldovan, Christine Clark, Mitchell Bowden, Jown Williams.
1 Adaptive Subjective Triggers for Opinionated Document Retrieval (WSDM 09’) Kazuhiro Seki, Kuniaki Uehara Date: 11/02/09 Speaker: Hsu, Yu-Wen Advisor:
Finding document topics for improving topic segmentation Source: ACL2007 Authors: Olivier Ferret (18 route du Panorama, BP6) Reporter:Yong-Xiang Chen.
Ranked Retrieval INST 734 Module 3 Doug Oard. Agenda Ranked retrieval  Similarity-based ranking Probability-based ranking.
Event-Based Extractive Summarization E. Filatova and V. Hatzivassiloglou Department of Computer Science Columbia University (ACL 2004)
Shallow & Deep QA Systems Ling 573 NLP Systems and Applications April 9, 2013.
(Pseudo)-Relevance Feedback & Passage Retrieval Ling573 NLP Systems & Applications April 28, 2011.
1 Question Answering and Logistics. 2 Class Logistics  Comments on proposals will be returned next week and may be available as early as Monday  Look.
1 ICASSP Paper Survey Presenter: Chen Yi-Ting. 2 Improved Spoken Document Retrieval With Dynamic Key Term Lexicon and Probabilistic Latent Semantic Analysis.
Analysis of Experiments on Hybridization of different approaches in mono and cross-language information retrieval DAEDALUS – Data, Decisions and Language,
Feature Assignment LBSC 878 February 22, 1999 Douglas W. Oard and Dagobert Soergel.
Integrating linguistic knowledge in passage retrieval for question answering J¨org Tiedemann Alfa Informatica, University of Groningen HLT/EMNLP 2005.
Question Answering Passage Retrieval Using Dependency Relations (SIGIR 2005) (National University of Singapore) Hang Cui, Renxu Sun, Keya Li, Min-Yen Kan,
1 Predicting Answer Location Using Shallow Semantic Analogical Reasoning in a Factoid Question Answering System Hapnes Toba, Mirna Adriani, and Ruli Manurung.
Short Text Similarity with Word Embedding Date: 2016/03/28 Author: Tom Kenter, Maarten de Rijke Source: CIKM’15 Advisor: Jia-Ling Koh Speaker: Chih-Hsuan.
Query Reformulation & Answer Extraction
Ling 573 Systems and Applications May 3, 2016
Entity- & Topic-Based Information Ordering
Improving a Pipeline Architecture for Shallow Discourse Parsing
ABDULLAH ALOTAYQ, DONG WANG, ED PHAM PROJECT BY:
Table Cell Search for Question Answering Huan Sun
Introduction to Machine Reading Comprehension
Introduction to Search Engines
Presentation transcript:

Passage Retrieval & Re-ranking Ling573 NLP Systems and Applications May 5, 2011

Reranking with Deeper Processing Passage Reranking for Question Answering Using Syntactic Structures and Answer Types Atkolga et al, 2011 Reranking of retrieved passages Integrates Syntactic alignment Answer type Named Entity information

Motivation Issues in shallow passage approaches: From Tellex et al.

Motivation Issues in shallow passage approaches: From Tellex et al. Retrieval match admits many possible answers Need answer type to restrict

Motivation Issues in shallow passage approaches: From Tellex et al. Retrieval match admits many possible answers Need answer type to restrict Question implies particular relations Use syntax to ensure

Motivation Issues in shallow passage approaches: From Tellex et al. Retrieval match admits many possible answers Need answer type to restrict Question implies particular relations Use syntax to ensure Joint strategy required Checking syntactic parallelism when no answer, useless Current approach incorporates all (plus NER)

Baseline Retrieval Bag-of-Words unigram retrieval (BOW)

Baseline Retrieval Bag-of-Words unigram retrieval (BOW) Question analysis: QuAn ngram retrieval, reformulation

Baseline Retrieval Bag-of-Words unigram retrieval (BOW) Question analysis: QuAn ngram retrieval, reformulation Question analysis + Wordnet: QuAn-Wnet Adds 10 synonyms of ngrams in QuAn

Baseline Retrieval Bag-of-Words unigram retrieval (BOW) Question analysis: QuAn ngram retrieval, reformulation Question analysis + Wordnet: QuAn-Wnet Adds 10 synonyms of ngrams in QuAn Best performance: QuAn-Wnet (baseline)

Dependency Information Assume dependency parses of questions, passages Passage = sentence Extract undirected dependency paths b/t words

Dependency Information Assume dependency parses of questions, passages Passage = sentence Extract undirected dependency paths b/t words Find path pairs between words (q k,a l ),(q r,a s ) Where q/a words ‘match’ Word match if a) same root or b) synonyms

Dependency Information Assume dependency parses of questions, passages Passage = sentence Extract undirected dependency paths b/t words Find path pairs between words (q k,a l ),(q r,a s ) Where q/a words ‘match’ Word match if a) same root or b) synonyms Later: require one pair to be question word/Answer term Train path ‘translation pair’ probabilities

Dependency Information Assume dependency parses of questions, passages Passage = sentence Extract undirected dependency paths b/t words Find path pairs between words (q k,a l ),(q r,a s ) Where q/a words ‘match’ Word match if a) same root or b) synonyms Later: require one pair to be question word/Answer term Train path ‘translation pair’ probabilities Use true Q/A pairs, GIZA++, IBM model 1 Yields Pr(label a,label q )

Dependency Path Similarity From Cui

Dependency Path Similarity

Similarity Dependency path matching

Similarity Dependency path matching Some paths match exactly Many paths have partial overlap or differ due to question/declarative contrasts

Similarity Dependency path matching Some paths match exactly Many paths have partial overlap or differ due to question/declarative contrasts Approaches have employed Exact match Fuzzy match Both can improve over baseline retrieval, fuzzy more

Dependency Path Similarity Cui et al scoring Sum over all possible paths in a QA candidate pair

Dependency Path Similarity Cui et al scoring Sum over all possible paths in a QA candidate pair

Dependency Path Similarity Cui et al scoring Sum over all possible paths in a QA candidate pair

Dependency Path Similarity Atype-DP Restrict first q,a word pair to Qword, ACand Where Acand has correct answer type by NER

Dependency Path Similarity Atype-DP Restrict first q,a word pair to Qword, ACand Where Acand has correct answer type by NER Sum over all possible paths in a QA candidate pair with best answer candidate

Dependency Path Similarity Atype-DP Restrict first q,a word pair to Qword, ACand Where Acand has correct answer type by NER Sum over all possible paths in a QA candidate pair with best answer candidate

Comparisons Atype-DP-IP Interpolates DP score with original retrieval score

Comparisons Atype-DP-IP Interpolates DP score with original retrieval score QuAn-Elim: Acts a passage answer-type filter Excludes any passage w/o correct answer type

Results Atype-DP-IP best

Results Atype-DP-IP best Raw dependency:‘brittle’; NE failure backs off to IP

Results Atype-DP-IP best Raw dependency:‘brittle’; NE failure backs off to IP QuAn-Elim: NOT significantly worse

Units of Retrieval Simple is Best: Experiments with Different Document Segmentation Strategies for Passage Retrieval Tiedemann and Mur, 2008 Comparison of units for retrieval in QA Documents Paragraphs Sentences Semantically-based units (discourse segments) Spans

Motivation Passage units necessary for QA Focused sources for answers Typically > 20 passage candidates yield poor QA Retrieval fundamentally crucial Re-ranking passages is hard Tellex et al experiments Improvements for passage reranking, but Still dramatically lower than oracle retrieval rates

Passages Some basic advantages for retrieval (vs documents) Documents vary in Length, Topic term density, Etc across type Passages can be less variable Effectively normalizing for length

What Makes a Passage? Sources of passage information Manual: Existing markup E.g., Sections, Paragraphs Issues: ? Still highly variable: Wikipedia vs Newswire Potentially ambiguous: blank lines separate ….. Not always available

What Makes a Passage? Automatic: Semantically motivated document segmentation Linguistic content Lexical patterns and relations Fixed length units: In words/chars or sentences/paragraphs Overlapping? Can be determined empirically All experiments use Zettair retrieval engine

Coreference Chains Coreference: NPs that refer to same entity Create an equivalence class Chains of coreference suggest entity-based coherence Passage: All sentences spanned by a coreference chain Can create overlapping passages Built with cluster-based ranking with own coref. System System has F-measure of 54.5%

TextTiling (Hearst) Automatic topic, sub-topic segmentation Computes similarity between neighboring text blocks Based on tf-idf weighted cosine similarity Compares similarity values Hypothesizes topic shift at dips b/t peaks in similarity Produces linear topic segmentation Existing implementations

Window-based Segmentation Fixed width windows: Based on words? Characters? Sentences? Sentences required for downstream deep processing Overlap? No overlap? No overlap is simple, but Not guaranteed to line up with natural boundaries Including document boundaries Overlap -> Sliding window

Evaluation Indexing and retrieval in Zettair system CLEF Dutch QA track Computes Lenient MRR measure Too few participants to assume pooling exhaustive Redundancy: Average # relevant passage per query Coverage: Proportion of Qs w/at least one relpass MAP Focus on MRR for prediction of end-to-end QA

Baselines Existing markup: Documents, paragraphs, sentences MRR-IR; MRR-QA (top 5); CLEF: end-to-end score Surprisingly good sentence results in top-5 and CLEF Sensitive to exact retrieval weighting

Semantic Passages Contrast: Sentence/coref: Sentences in coref. chains -> too long Bounded length Paragraphs and coref chains (bounded) TextTiling (CPAN) – Best : beats baseline

Fixed Size Windows Different lengths: non-overlapping 2-, 4-sentence units improve over semantic units

Sliding Windows Fixed length windows, overlapping Best MRR-QA values Small units with overlap Other settings weaker

Observations Competing retrieval demands: IR performance vs QA performance MRR at 5 favors: Small, fixed width units Advantageous for downstream processing too Any benefit of more sophisticated segments Outweighed by increased processing