Download presentation
Presentation is loading. Please wait.
Published byCortez MacPherson Modified over 10 years ago
1
Group 3 Chad Mills Esad Suskic Wee Teck Tan
2
Outline System and Data Document Retrieval Passage Retrieval Results Conclusion
3
System and Data DevelopmentTesting TREC 2004 TREC 2005 System: Indri http://www.lemurproject.org/ http://www.lemurproject.org/ Data:
4
Document Retrieval Baseline: Remove “?” Add Target String MAP: 0.307
5
Document Retrieval Attempted Improvement 1: Settings From Baseline Rewrite “When was…” questions as “[target] was [last word] on” queries MAP: 0.301 Best so far: 0.307
6
Document Retrieval Attempted Improvement 2: Settings From Baseline Remove “Wh” words Remove Stop Words Replaced Pronoun with Target String MAP: 0.319 Best so far: 0.307 “Wh” / Stop WordsWhat, Who, Where, Why, How many, How often, How long, Which, How did, Does, is, the, a, an, of, was, as Pronounhe, she, it, its, they, their, his
7
Document Retrieval Attempted Improvement 3: Settings From Improvement 2 Index Stemmed (Krovetz Stemmer) MAP: 0.336 Best so far: 0.319
8
Document Retrieval Attempted Improvement 4: Settings From Improvement 3 Remove Punctuations Remove Non Alphanumeric Characters MAP: 0.374 Best so far: 0.336
9
Document Retrieval Attempted Improvement 5: Settings From Improvement 4 Remove Duplicate Words MAP: 0.377 Best so far: 0.374
10
Passage Retrieval Baseline: Out-of-the-box Indri Same Question Formulation Changed “#combine(“ to “#combine[passageX:Y](” Passage Window, Top 20, No Re-ranking X=40 Y=20 Strict0.126 Lenient0.337 X=200 Y=100 Strict0.414 Lenient0.537
11
Passage Retrieval Attempted Re-ranking Mallet MaxEnt Classifier Training Set TREC 2004 ○ 80% Train : 20% Dev ○ Split by Target ○ Avoid Cheating e.g. Question 1.* all in either Train or Dev Labels: + Passage has Correct Answer - Passage doesn’t have Answer
12
Passage Retrieval Features used: For both Passage and Question+Target: ○ unigram, bigram, trigram ○ POS tags – unigram, bigram, trigram Question/Passage Correspondence: ○ # of Overlapping Terms (and bigrams) ○ Distance between Overlapping Terms Tried Top 20 Passages from Indri, and Expanding to Top 200 Passages
13
Passage Retrieval Result: all attempts were worse than before Example confusion matrix: Many negative examples, 67-69% accurate on all feature combinations tried +- +16267 -37620
14
Indri was very good to start with E.g. Q10.1 Passage Re-Ranking Indri RankHas Answer 1Yes 2No 3Yes 4 5No Our RankHas AnswerP(Yes)P(No)Indri Rank 1No0.0760.9245 2No0.0270.97318 3Yes0.0140.9868 4No0.0110.9897 5Yes0.0070.99314 Our first 2 were wrong, only 1 of Indri’s top 5 in our top 5 If completely replacing rank, must be very good Many low confidence scores (e.g. 7.6% P(Yes) was best) Slight edit to Indri ranking less bad, but no good system found E.g. bump high-confidence Yes to top of list, leave others in Indri order
15
Results TREC 2004: TREC 2005: MAP0.377 Strict MRR0.414 Lenient MRR0.537 MAP0.316 Strict MRR0.366 Lenient MRR0.543
16
References Fang – “A Re-examination of Query Expansion Using Lexical Resources” Tellex – “Quantitative Evaluation of Passage Retrieval Algorithms for Question Answering”
17
Conclusions Cleaned Input Small Targeted Stop Word List Minimal Setting Indri Performs PR Well OOTB Re-ranking Implementation Needs to be Really Good Feature Selection didn’t Help Slight Adjustment Instead of Whole Different Ranking Might Help
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.