1. Group 3 Chad Mills Esad Suskic Wee Teck Tan

2. Outline  System and Data  Document Retrieval  Passage Retrieval  Results  Conclusion

3. System and Data DevelopmentTesting TREC 2004 TREC 2005  System: Indri http://www.lemurproject.org/ http://www.lemurproject.org/  Data:

4. Document Retrieval  Baseline: Remove “?” Add Target String  MAP: 0.307

5. Document Retrieval  Attempted Improvement 1: Settings From Baseline Rewrite “When was…” questions as “[target] was [last word] on” queries  MAP: 0.301 Best so far: 0.307

6. Document Retrieval  Attempted Improvement 2: Settings From Baseline Remove “Wh” words Remove Stop Words Replaced Pronoun with Target String  MAP: 0.319 Best so far: 0.307 “Wh” / Stop WordsWhat, Who, Where, Why, How many, How often, How long, Which, How did, Does, is, the, a, an, of, was, as Pronounhe, she, it, its, they, their, his

7. Document Retrieval  Attempted Improvement 3: Settings From Improvement 2 Index Stemmed (Krovetz Stemmer)  MAP: 0.336 Best so far: 0.319

8. Document Retrieval  Attempted Improvement 4: Settings From Improvement 3 Remove Punctuations Remove Non Alphanumeric Characters  MAP: 0.374 Best so far: 0.336

9. Document Retrieval  Attempted Improvement 5: Settings From Improvement 4 Remove Duplicate Words  MAP: 0.377 Best so far: 0.374

10. Passage Retrieval  Baseline: Out-of-the-box Indri Same Question Formulation Changed “#combine(“ to “#combine[passageX:Y](” Passage Window, Top 20, No Re-ranking X=40 Y=20 Strict0.126 Lenient0.337 X=200 Y=100 Strict0.414 Lenient0.537

11. Passage Retrieval  Attempted Re-ranking Mallet MaxEnt Classifier Training Set TREC 2004 ○ 80% Train : 20% Dev ○ Split by Target ○ Avoid Cheating e.g. Question 1.* all in either Train or Dev Labels: + Passage has Correct Answer - Passage doesn’t have Answer

12. Passage Retrieval  Features used: For both Passage and Question+Target: ○ unigram, bigram, trigram ○ POS tags – unigram, bigram, trigram Question/Passage Correspondence: ○ # of Overlapping Terms (and bigrams) ○ Distance between Overlapping Terms  Tried Top 20 Passages from Indri, and Expanding to Top 200 Passages

13. Passage Retrieval  Result: all attempts were worse than before Example confusion matrix:  Many negative examples, 67-69% accurate on all feature combinations tried +- +16267 -37620

14.  Indri was very good to start with  E.g. Q10.1 Passage Re-Ranking Indri RankHas Answer 1Yes 2No 3Yes 4 5No Our RankHas AnswerP(Yes)P(No)Indri Rank 1No0.0760.9245 2No0.0270.97318 3Yes0.0140.9868 4No0.0110.9897 5Yes0.0070.99314  Our first 2 were wrong, only 1 of Indri’s top 5 in our top 5  If completely replacing rank, must be very good  Many low confidence scores (e.g. 7.6% P(Yes) was best)  Slight edit to Indri ranking less bad, but no good system found  E.g. bump high-confidence Yes to top of list, leave others in Indri order

15. Results  TREC 2004:  TREC 2005: MAP0.377 Strict MRR0.414 Lenient MRR0.537 MAP0.316 Strict MRR0.366 Lenient MRR0.543

16. References  Fang – “A Re-examination of Query Expansion Using Lexical Resources”  Tellex – “Quantitative Evaluation of Passage Retrieval Algorithms for Question Answering”

17. Conclusions  Cleaned Input  Small Targeted Stop Word List  Minimal Setting  Indri Performs PR Well OOTB  Re-ranking Implementation Needs to be Really Good Feature Selection didn’t Help Slight Adjustment Instead of Whole Different Ranking Might Help