Automatic Set Expansion for List Question Answering Richard C. Wang, Nico Schlaefer, William W. Cohen, and Eric Nyberg Language Technologies Institute Carnegie Mellon University Pittsburgh, PA USA

Language Technologies Institute, Carnegie Mellon University Richard C. Wang 2 / 30 Set Expansion for List Question Answering Task Automatically improve answers generated by Question Answering systems for list questions, by using a Set Expansion system. For example:  Name cities that have Starbucks. QA AnswersExpanded Answers Boston Seattle Carnegie-Mellon Aquafina Google Logitech Seattle Boston Chicago Pittsburgh Carnegie-Mellon Google Better!

Language Technologies Institute, Carnegie Mellon University Richard C. Wang 3 / 30 Set Expansion for List Question Answering Outline Introduction  Question Answering  Set Expansion Proposed Approach  Aggressive Fetcher  Lenient Extractor  Hinted Expander Experimental Results  QA System: Ephyra  Other QA Systems Conclusion

Language Technologies Institute, Carnegie Mellon University Richard C. Wang 4 / 30 Set Expansion for List Question Answering Question Answering (QA) Question Answering task:  Retrieve answers to natural language questions Different question types:  Factoid questions  List questions  Definitional questions  Opinion questions Major QA evaluations:  Text REtrieval Conference (TREC): English  NTCIR: Japanese, Chinese  CLEF: European languages

Language Technologies Institute, Carnegie Mellon University Richard C. Wang 5 / 30 Set Expansion for List Question Answering Typical QA Pipeline Question Analysis Query Generation & Search Candidate Generation Answer Scoring Knowledge Sources Question String Analyzed Question Search Results Candidate Answers Scored Answers The two original text smileys were invented on September 19, 1982 by Scott E. Fahlman... smileys September 19, 1982 Scott E. Fahlman CandidateScore Scott E. Fahlman smileys0.418 September 19, “Who invented the smiley?” Answer type: Person Keywords: invented, smiley...

Language Technologies Institute, Carnegie Mellon University Richard C. Wang 6 / 30 Set Expansion for List Question Answering QA System: Ephyra (Schlaefer et al., TREC 2007) History:  Developed at University of Karlsruhe, Germany and Carnegie Mellon University, USA  TREC participations in 2006 (13 th out of 27 teams) and 2007 (7 th out of 21 teams)  Released into open source in 2008 Different candidate generators:  Answer type classification  Regular expression matching  Semantic parsing Available for download at:

Language Technologies Institute, Carnegie Mellon University Richard C. Wang 7 / 30 Set Expansion for List Question Answering Outline Introduction  Question Answering  Set Expansion Proposed Approach  Aggressive Fetcher  Lenient Extractor  Hinted Expander Experimental Results  QA System: Ephyra  Other QA Systems Conclusion

Language Technologies Institute, Carnegie Mellon University Richard C. Wang 8 / 30 Set Expansion for List Question Answering Set Expansion (SE) For example,  Given a query: {“survivor”, “amazing race”}  Answer is: {“american idol”, “big brother”,....} More formally,  Given a small number of seeds: x 1, x 2, …, x k where each x i S t  Answer is a listing of other probable elements: e 1, e 2, …, e n where each e i S t A well-known example of a web-based set expansion system is Google Sets™ 

Language Technologies Institute, Carnegie Mellon University Richard C. Wang 9 / 30 Set Expansion for List Question Answering SE System: SEAL (Wang & Cohen, ICDM 2007) Features  Independent of human/markup language Support seeds in English, Chinese, Japanese, Korean,... Accept documents in HTML, XML, SGML, TeX, WikiML, …  Does not require pre-annotated training data Utilize readily-available corpus: World Wide Web Based on two research contributions  Automatically construct wrappers for extracting candidate items  Rank extracted items using random graph walk Try it out for yourself:

Language Technologies Institute, Carnegie Mellon University Richard C. Wang 10 / 30 Set Expansion for List Question Answering SEAL’s SE Pipeline Fetcher: downloads web pages from the Web Extractor: learns wrappers from web pages Ranker: ranks entities extracted by wrappers Canon Nikon Olympus Pentax Sony Kodak Minolta Panasonic Casio Leica Fuji Samsung …

Language Technologies Institute, Carnegie Mellon University Richard C. Wang 11 / 30 Set Expansion for List Question Answering Challenge SE systems require relevant (non-noisy) seeds, but answers produced by QA systems are often noisy. How can we integrate those two systems together?  We propose three extensions to SEAL Aggressive Fetcher Lenient Extractor Hinted Expander

Language Technologies Institute, Carnegie Mellon University Richard C. Wang 12 / 30 Set Expansion for List Question Answering Outline Introduction  Question Answering  Set Expansion Proposed Approach  Aggressive Fetcher  Lenient Extractor  Hinted Expander Experimental Results  QA System: Ephyra  Other QA Systems Conclusion

Language Technologies Institute, Carnegie Mellon University Richard C. Wang 13 / 30 Set Expansion for List Question Answering Original Fetcher Procedure: 1. Compose a search query by concatenating all seeds 2. Use Google to request top 100 web pages 3. Fetch web pages and send to the Extractor Seeds Boston Seattle Carnegie-Mellon Query Boston Seattle Carnegie-Mellon

Language Technologies Institute, Carnegie Mellon University Richard C. Wang 14 / 30 Set Expansion for List Question Answering Proposed Fetcher Aggressive Fetcher (AF)  Sends a two-seed query for every possible pair of seeds to the search engines  More likely to compose queries containing only relevant seeds Seeds Boston Seattle Carnegie-Mellon Queries Boston Seattle Boston Carnegie-Mellon Seattle Carnegie-Mellon

Language Technologies Institute, Carnegie Mellon University Richard C. Wang 15 / 30 Set Expansion for List Question Answering Outline Introduction  Question Answering  Set Expansion Proposed Approach  Aggressive Fetcher  Lenient Extractor  Hinted Expander Experimental Results  QA System: Ephyra  Other QA Systems Conclusion

Language Technologies Institute, Carnegie Mellon University Richard C. Wang 16 / 30 Set Expansion for List Question Answering Original Extractor A wrapper is a pair of L and R context string  Maximally-long contextual strings that bracket at least one instance of every seed  Extracts strings between L and R Learn wrappers from web pages and seeds on the fly  Utilize semi-structured documents  Wrappers defined at character level No tokenization required (language-independent) However, very page specific (page-dependent)

Language Technologies Institute, Carnegie Mellon University Richard C. Wang 17 / 30 Set Expansion for List Question Answering

Language Technologies Institute, Carnegie Mellon University Richard C. Wang 18 / 30 Set Expansion for List Question Answering Proposed Extractor Lenient Extractor (LE)  Maximally-long contextual strings that bracket at least one instance of a minimum of two seeds  More likely to find useful contexts that bracket only relevant seeds Text... in Boston City Hall in Seattle City Hall at Boston University at Seattle University at Carnegie-Mellon University... Learned Wrapper (w/o LE) at University Learned Wrappers (w/ LE) at University in City Hall

Language Technologies Institute, Carnegie Mellon University Richard C. Wang 19 / 30 Set Expansion for List Question Answering Outline Introduction  Question Answering  Set Expansion Proposed Approach  Aggressive Fetcher  Lenient Extractor  Hinted Expander Experimental Results  QA System: Ephyra  Other QA Systems Conclusion

Language Technologies Institute, Carnegie Mellon University Richard C. Wang 20 / 30 Set Expansion for List Question Answering Hinted Expander (HE) Utilizes contexts in the question to constrain SEAL’s search space on the Web  Extract up to three keywords from the question using Ephyra’s keyword extractor  Append the keywords to the search query Example:  Name cities that have Starbucks. More likely to find documents containing desired set of answers

Language Technologies Institute, Carnegie Mellon University Richard C. Wang 21 / 30 Set Expansion for List Question Answering Outline Introduction  Question Answering  Set Expansion Proposed Approach  Aggressive Fetcher  Lenient Extractor  Hinted Expander Experimental Results  QA System: Ephyra  Other QA Systems Conclusion

Language Technologies Institute, Carnegie Mellon University Richard C. Wang 22 / 30 Set Expansion for List Question Answering Experiment #1: Ephyra Evaluate on TREC 13, 14, and 15 datasets  55, 93, and 89 list questions respectively Use SEAL to expand top four answers from Ephyra  Outputs a list of answers ranked by confidence scores For each dataset, we report:  Mean Average Precision (MAP) Mean of average precision for each ranked list  Average F 1 with Optimal Per-Question Threshold For each question, cut off the list at a threshold which maximizes the F 1 score for that particular question

Language Technologies Institute, Carnegie Mellon University Richard C. Wang 23 / 30 Set Expansion for List Question Answering Experiment #1: Ephyra

Language Technologies Institute, Carnegie Mellon University Richard C. Wang 24 / 30 Set Expansion for List Question Answering Experiment #2: Ephyra In practice, thresholds are unknown For each dataset, do 5-fold cross validation:  Train: Find one optimal threshold for four folds  Test: Use the threshold to evaluate the fifth fold Introduce a fourth dataset: All  Union of TREC 13, 14, and 15 Introduce another system: Hybrid  Intersection of original answers from Ephyra and expanded answers from SEAL

Language Technologies Institute, Carnegie Mellon University Richard C. Wang 25 / 30 Set Expansion for List Question Answering Experiment #2: Ephyra

Language Technologies Institute, Carnegie Mellon University Richard C. Wang 26 / 30 Set Expansion for List Question Answering Outline Introduction  Question Answering  Set Expansion Proposed Approach  Aggressive Fetcher  Lenient Extractor  Hinted Expander Experimental Results  QA System: Ephyra  Other QA Systems Conclusion

Language Technologies Institute, Carnegie Mellon University Richard C. Wang 27 / 30 Set Expansion for List Question Answering Experiment: Other QA Systems Top five QA systems that perform the best on list questions in TREC 15 evaluation 1. Language Computer Corporation (lccPA06) 2. The Chinese University of Hong Kong (cuhkqaepisto) 3. National University of Singapore (NUSCHUAQA1) 4. Fudan University (FDUQAT15A) 5. National Security Agency (QACTIS06C) For each QA system, train thresholds for SEAL and Hybrid on the union of TREC 13 and 14  Expand top four answers from the QA systems on TREC 15, and apply the trained threshold

Language Technologies Institute, Carnegie Mellon University Richard C. Wang 28 / 30 Set Expansion for List Question Answering Experiment: Top QA Systems

Language Technologies Institute, Carnegie Mellon University Richard C. Wang 29 / 30 Set Expansion for List Question Answering Conclusion A feasible method for integrating a SE approach into any QA system Proposed SE approach is effective  Improves QA systems on list questions by using only a few top answers as seeds Proposed hybrid system is effective  Improves Ephyra and (most) top five QA systems

Language Technologies Institute, Carnegie Mellon University Richard C. Wang 30 / 30 Set Expansion for List Question Answering Thank You!