1. Wei Xu, Ralph Grishman, Le Zhao (CMU) xuwei@cs.nyu.edu New York University Novmember 24, 2011

2.  New York University (since 2007)  Tsinghua University  Microsoft Research  Amazon.com  ETS  France Telecom  Hong Kong Polytechnic University  Information Retrieval  Information Extraction  Spelling Correction  Summarization

3.  Knowledge Base Population (KBP) slot filling task (Ji et al. 2010) at TAC 2010:  find attributes of a specified person or organization from a large corpus person:employee_of organization:headquarters Etc.

4.  No time to perform in-depth extraction  Little annotated data  Blend QA and IR with IE

5.  Like QA, begin with passage retrieval to find passages relevant to the person or organization and its attributes  Unlike QA, there is a fixed set of relations which allows more specialized learning methods

6.  Passage Retrieval  Hand-selected keywords (Surdeanu et al. 2010, Nguye et al. 2010)  Expected named entity types (Chrupala et al. 2010)  Relevance based on the similarity between terms (Fang et al. 2010)

7.  Distant Supervision  Use noisy training data generated automatically from a related, but different, type of dataset to solve problem on another type of data  Rich-featured logistic regression model (Mintz et al. 2009)

8.  Use Lemur to retrieve passages containing the given entity  Re-rank the retrieved passages preferring those that contain:  the target entity or pronouns  named entity of the expected type  relevant terms of the target relation

9.  Baseline: hand selected keywords + 635 common title words (person-employer)  IKFB: weighted keyword list learned by distant supervision

10. FREEBASE Steve Jobs – Apple Inc. FREEBASE Steve Jobs – Apple Inc. WIKIPEDIA – Steve Jobs * “He is co-founder, chairman, and former chief executive officer of Apple Inc.” * “On August 24, 2011, Jobs announced his resignation from his role as Apple's CEO.” * “In March 1998, to concentrate Apple's efforts on returning to profitability, Jobs terminated a number of projects, such as Newton, Cyberdog, and OpenDoc.” * “Even though Jobs earned only $1 a year as CEO of Apple, he holds 5.426 million Apple shares, as well as 138 million shares in Disney (which he had received in exchange for Disney's acquisition of Pixar).” WIKIPEDIA – Steve Jobs * “He is co-founder, chairman, and former chief executive officer of Apple Inc.” * “On August 24, 2011, Jobs announced his resignation from his role as Apple's CEO.” * “In March 1998, to concentrate Apple's efforts on returning to profitability, Jobs terminated a number of projects, such as Newton, Cyberdog, and OpenDoc.” * “Even though Jobs earned only $1 a year as CEO of Apple, he holds 5.426 million Apple shares, as well as 138 million shares in Disney (which he had received in exchange for Disney's acquisition of Pixar).”

11. person - employer facultycurrentlyheadquarter shareholder professorchairman sinceat emeritusjoin housemasterchain chancellorfounder dynamoposition teach chief yesterdaysupermarket executive departmentretailpsychology dean conglomerateofficercreative rector headmergerassociate lecturer subsidiarydirectorchair therapeuticscompanypresidentteaching

12. KBP 2010 training data human annotated, incomplete keys 1.8 million documents 67 person entities 54 instances of person-employer

13. 10702 person:employee_of in Freebase 4497 are found in Wikipedia articles 6574 positive 93756 negative sentences 2436 indicative keywords

14.  Coverage: the proportion of relation instances that can be found within top n retrieved passages  For 31.5% of relation instances, only one passage had to be examined using IKFB, 16.7% for baseline, 9.3% for Lemur

15.  Redundancy: the average number of passages within the top n ranks that contain a correct answer  18.7% of the top 10 ranked passages by IKFB contain correct answers, 5.93% for baseline

16.  Employer of ‘John Dewey’ – 1 st Ranked Passage  An institution that sees itself as an uncon-ventional alternative to other colleges, the New School was founded in 1919 by a group of pro-fessors, including the philosopher and education reformer John Dewey, who had resigned in pro-test from Columbia. They could not abide by a stance taken by Columbia's president at the time, Nicholas Murray Butler, that faculty members had to support America's entry into World War I.