1. LING 573 Deliverable 3 Jonggun Park Haotian He Maria Antoniak Ron Lockwood

2. Closed Class filters 14 Animals, colors, companies, continents, countries, sports team, languages, occupations, periodic table, race, us-cities, us-presidents, us-states, and us- universities.

3. Query EXPAANSSION!

4. Query Expansion Who is the president of the United States? President united states nations council How long did it take to build the Tower of Pisa? long build tower pisa women’s station

5. Question Classification Software package: Mallet Classification algorithms: MaxEnt, NaiveBayes, Winnow, DecisionTree Training Data: - TREC-2004.xml - Training set 5 (5500 labeled questions) (Li & Roth) Test Data: - TREC-2005.xml - Testing set (Li & Roth)

6. Feature selection: - Unigram - Bigram - Trigram - Question word - NER tags

7. Conclusion: Maximum accuracy: TREC-2005 as test file: 0.8535911602209945 - MaxEnt, Unigram + Bigram + Wh-words TREC-10 as test file: 0.854 - MaxEnt, Unigram + Bigram + Wh-words

8. Other findings: 1.Trigram does not helps and drags the accuracy down. 2.NER feature does not helps and causes a slight drop-down.

9. Web Boosting Resources: jsoup, Bing.com Query: original question + target string Results: top 50 web snippets, stored in a text file

10. Web Boosting Challenges and Successes Which search engine or answer website to use? How to avoid throttling? How to integrate results into our system? How to edit results to make them more useful for our answer ranking system?

11. Main Changes Use web query as input to the redundancy-based answer extraction engine This replaces our paragraph based index Answer type classification now feeds into answer extraction Filtering of candidate answers by answer type in combination with NER on the answers Following types are handled: NUM, LOC, HUM, ENTY

12. Main changes (continued) Filtering of closed class questions using lists E.g. pro sports teams, colors, etc. Filtering out of terms with occurrences in less than 2 snippets Return 250 char. answer instead of 1-4 words

13. Answer Extraction Details Input to the Extraction Engine Query word list Stop-word list Focus-word list (e.g. meters, liters, miles, etc.) Passage list – the paragraph results of the query 1.N-gram generation and occurrence counting 2.Filtering out stop words and query words 3.Filter by answer type

14. Answer Extraction Details 4. Combining unigram counts with n-gram counts 5. Weighting candidates with idf scores 6. Re-rank candidates 1.Eliminate ones that don’t have evidence in at least 2 snippets 2.Eliminate ones that don’t match a closed class list (for certain questions.) 7. Verifying candidates in documents 1.Use bag of words query from the candidate sub- snippet + query words against Lucene index

15. Results D2: strict = 0.01 lenient = 0.064 D3: strict = 0.133 lenient = 0.371