1. Answering List Questions using Co-occurrence and Clustering Majid Razmara and Leila Kosseim Concordia University m_razma@cs.concordia.ca

2. 2 Introduction Question Answering TREC QA track  Question Series  Corpora Target: American Girl dolls  FACTOID: In what year were American Girl dolls first introduced?  LIST: Name the historical dolls.  LIST : Which American Girl dolls have had TV movies made about them?  FACTOID: How much does an American Girl doll cost?  FACTOID: How many American Girl dolls have been sold?  FACTOID: What is the name of the American Girl store in New York?  FACTOID: What corporation owns the American Girl company?  OTHER: Other

3. 3 Hypothesis Answer Instances 1.Have the same semantic entity class 2.Co-occur within sentences, or 3.Occur in different sentences sharing similar context  Based on Distributional Hypothesis: “Words occurring in the same contexts tend to have similar meanings” [Harris, 1954].

4. Ltw_Eng_20050712.0032 (AQUAINT-2) United, which operates a hub at Dulles, has six luggage screening machines in its basement and several upstairs in the ticket counter area. Delta, Northwest, American, British Airways and KLM share four screening machines in the basement. Ltw_Eng_20060102.0106 (AQUAINT-2) Independence said its last flight Thursday will leave White Plains, N.Y., bound for Dulles Airport. Flyi suffered from rising jet fuel costs and the aggressive response of competitors, led by United and US Airways. New York Times (Web) Continental Airlines sued United Airlines and the committee that oversees operations at Washington Dulles International Airport yesterday, contending that recently installed baggage-sizing templates inhibited competition. Wikipedia (Web) AirTran At its peak of 600 flights daily, Independence, combined with service from JetBlue and AirTran, briefly made Dulles the largest low-cost hub in the United States. Target 232: "Dulles Airport“ Question 232.6: "Which airlines use Dulles” 4

5. 5 Our Approach 1. Create an initial candidate list  Answer Type Recognition  Document Retrieval  Candidate Answer Extraction  It may also be imported from an external source (e.g. Factoid QA) 2. Extract co-occurrence information 3. Cluster candidates based on their co- occurrence

6. 6 Answer Type Recognition 9 Types:  Person, Country, Organization, Job, Movie, Nationality, City, State, and Other Lexical Patterns  ^ (Name | List | What | Which) (persons | people | men | women | players | contestants | artists | opponents | students)  PERSON  ^ (Name | List | What | Which) (countries | nations)  COUNTRY Syntagmatic Patterns for Other types  ^ (WDT | WP | VB | NN) (DT | JJ)* (NNS | NNP | NN | JJ | )* (NNS | NNP | NN | NNPS) (VBN | VBD | VBZ | WP | $)  ^ (WDT | WP | VB | NN) (VBD | VBP) (DT | JJ | JJR | PRP$ | IN)* (NNS | NNP | NN | )* (NNS | NNP | NN) Type Resolution

7. 7 Resolves the answer subtype to one of the main types  List previous conductors of the Boston Pops.  Type: OTHER Sub Type: Conductor  PERSON WordNet's Hypernym Hierarchy

8. 8 Document Retrieval Document Collection  Source Document Collection  Few documents  To extract candidates  Domain Document Collection  Many documents  To extract co-occurrence information Query Generation  Google Query on Web  Lucene Query on Corpora Source Domain

9. 9 Candidate Answer Extraction Term Extraction  Extract all terms that conform to the expected answer type  Person, Organization, Job  Intersection of several NE taggers: LingPipe, Stanford tagger & Gate NE  To get a better precision  Country, State, City, Nationality  Gazetteer  To get a better precision  Movie, Other  Capitalized and quoted terms  Verification of Movie  Verification of Other numHits(“SubType Term” OR “Term SubType”) numHits(“Term”) numHits(GoogleQuery intitle:Term site:www.imdb.com)

10. 10 Co-occurrence Information Extraction Domain Collection Documents are split into sentences Each sentence is checked as to whether it contains candidate answers

11. Steps: 1.Put each candidate term t i in a separate cluster C i 2.Compute the similarity between each pair of clusters  Average Linkage 3.Merge two clusters with highest inter-cluster similarity 4.Update all relations between this new cluster and other clusters 5.Go to step 3 until  There are only N clusters, or  The similarity is less than a threshold 11 Hierarchical Agglomerative Clustering

12. Similarity between each pair of candidates Based on co-occurrence within sentences Using chi-square (  2 ) Shortcoming 12 The Similarity Measure Total  term i term i O 11 + O 21 O 21 O 11 term j O 12 + O 22 O 22 O 12  term j NO 21 + O 22 O 11 + O 12 Total

13. 13 Pinpointing the Right Cluster Question and target keywords are used as “spies” Spies are:  Inserted into the list of candidate answers  Are treated as candidate answers, hence  their similarity to one another and similarity to candidate answers are computed  they are clustered along with candidate answers The cluster with the most number of spies is returned  The spies are removed Other approaches

14. oman Cluster-31 Cluster-2Cluster-9 spain, bangladesh, japan, germany, haiti, nepal, china, sweden, iran, mexico, vietnam, belgium, lebanon, iraq, russia, turkey pakistan, 2005, afghanistan, octob, u.s, india, affect, earthquak pakistan pakistan, 2005, afghanistan, octob, u.s, india, affect, earthquak pakistan, 2005, afghanistan, octob, u.s, india, affect, earthquak Pakistan earthquakes of October 2005 Target 269: Pakistan earthquakes of October 2005 What countries were affected by this earthquake? Question 269.2: What countries were affected by this earthquake? Recall = 2/3 Precision = 2/3 F-score = 2/3 14

15. 15 0.479Best 0.085Median 0.000Worst F=14.5 Results in TREC 2007

16. 16 Evaluation of Clustering Baseline  List of candidate answers prior to clustering Our Approach  List of candidate answers filtered by the clustering Theoretical Maximum  The best possible output of clustering based on the initial list F-scoreRecallPrecisionQuestionsCorpus 0.098 0.154 0.472 0.407 0.287 0.407 0.064 0.141 1 237 2004 TREC 2005 2006 Baseline Our Approach Theoretical Max 0.106 0.163 0.485 0.388 0.248 0.388 0.075 0.165 1 85TREC 2007 Baseline Our Approach Theoretical Max

17. 17 Evaluation of each Question Type

18. 18 Future Work Developing a module that verifies whether each candidate is a member of the answer type  In case of Movie and Other types Using co-occurrence at the paragraph level rather than the sentence level  Anaphora Resolution can be used  Another method for similarity measure   2 does not work well with sparse data  for example, using Yates correction for continuity (Yates’  2 ) Using different clustering approaches Using different similarity measures  Mutual Information

19. 19 Questions?