Ang Sun Ralph Grishman Wei Xu Bonan Min November 15, 2011 TAC 2011 Workshop Gaithersburg, Maryland USA.

Slides:



Advertisements
Similar presentations
Pseudo-Relevance Feedback For Multimedia Retrieval By Rong Yan, Alexander G. and Rong Jin Mwangi S. Kariuki
Advertisements

Arnd Christian König Venkatesh Ganti Rares Vernica Microsoft Research Entity Categorization Over Large Document Collections.
Multi-Document Person Name Resolution Michael Ben Fleischman (MIT), Eduard Hovy (USC) From Proceedings of ACL-42 Reference Resolution workshop 2004.
1 Evaluation Rong Jin. 2 Evaluation  Evaluation is key to building effective and efficient search engines usually carried out in controlled experiments.
Overview of the TAC2013 Knowledge Base Population Evaluation: Temporal Slot Filling Mihai Surdeanu with a lot help from: Hoa Dang, Joe Ellis, Heng Ji,
Overview of the TAC2013 Knowledge Base Population Evaluation: English Slot Filling Mihai Surdeanu with a lot help from: Hoa Dang, Joe Ellis, Heng Ji, and.
Linked data: P redicting missing properties Klemen Simonic, Jan Rupnik, Primoz Skraba {klemen.simonic, jan.rupnik,
Statistical Machine Translation Part II – Word Alignments and EM Alex Fraser Institute for Natural Language Processing University of Stuttgart
Supervised Learning Techniques over Twitter Data Kleisarchaki Sofia.
Beyond TREC-QA Ling573 NLP Systems and Applications May 28, 2013.
Overview of the KBP 2013 Slot Filler Validation Track Hoa Trang Dang National Institute of Standards and Technology.
Mining External Resources for Biomedical IE Why, How, What Malvina Nissim
The Impact of Task and Corpus on Event Extraction Systems Ralph Grishman New York University Malta, May 2010 NYU.
NYU ANLP-00 1 Automatic Discovery of Scenario-Level Patterns for Information Extraction Roman Yangarber Ralph Grishman Pasi Tapanainen Silja Huttunen.
Exploiting Dictionaries in Named Entity Extraction: Combining Semi-Markov Extraction Processes and Data Integration Methods William W. Cohen, Sunita Sarawagi.
Multiple Criteria for Evaluating Land Cover Classification Algorithms Summary of a paper by R.S. DeFries and Jonathan Cheung-Wai Chan April, 2000 Remote.
Evaluating Search Engine
The use of unlabeled data to improve supervised learning for text summarization MR Amini, P Gallinari (SIGIR 2002) Slides prepared by Jon Elsas for the.
Object retrieval with large vocabularies and fast spatial matching
Event Extraction: Learning from Corpora Prepared by Ralph Grishman Based on research and slides by Roman Yangarber NYU.
Methods for Domain-Independent Information Extraction from the Web An Experimental Comparison Oren Etzioni et al. Prepared by Ang Sun
Co-training LING 572 Fei Xia 02/21/06. Overview Proposed by Blum and Mitchell (1998) Important work: –(Nigam and Ghani, 2000) –(Goldman and Zhou, 2000)
Ranking by Odds Ratio A Probability Model Approach let be a Boolean random variable: document d is relevant to query q otherwise Consider document d as.
Learning syntactic patterns for automatic hypernym discovery Rion Snow, Daniel Jurafsky and Andrew Y. Ng Prepared by Ang Sun
(Some issues in) Text Ranking. Recall General Framework Crawl – Use XML structure – Follow links to get new pages Retrieve relevant documents – Today.
Active Learning for Class Imbalance Problem
Empirical Methods in Information Extraction Claire Cardie Appeared in AI Magazine, 18:4, Summarized by Seong-Bae Park.
Reyyan Yeniterzi Weakly-Supervised Discovery of Named Entities Using Web Search Queries Marius Pasca Google CIKM 2007.
1 A Bayesian Method for Guessing the Extreme Values in a Data Set Mingxi Wu, Chris Jermaine University of Florida September 2007.
Overview of the KBP 2012 Slot-Filling Tasks Hoa Trang Dang (National Institute of Standards and Technology Javier Artiles (Rakuten Institute of Technology)
Classifier Evaluation Vasileios Hatzivassiloglou University of Texas at Dallas.
1 Efficient Search Ranking in Social Network ACM CIKM2007 Monique V. Vieira, Bruno M. Fonseca, Rodrigo Damazio, Paulo B. Golgher, Davi de Castro Reis,
Querying Structured Text in an XML Database By Xuemei Luo.
Wei Xu, Ralph Grishman, Le Zhao (CMU) New York University Novmember 24, 2011.
Partially Supervised Classification of Text Documents by Bing Liu, Philip Yu, and Xiaoli Li Presented by: Rick Knowles 7 April 2005.
1 Automating Slot Filling Validation to Assist Human Assessment Suzanne Tamang and Heng Ji Computer Science Department and Linguistics Department, Queens.
Detecting Dominant Locations from Search Queries Lee Wang, Chuang Wang, Xing Xie, Josh Forman, Yansheng Lu, Wei-Ying Ma, Ying Li SIGIR 2005.
Distributed Information Retrieval Server Ranking for Distributed Text Retrieval Systems on the Internet B. Yuwono and D. Lee Siemens TREC-4 Report: Further.
Collocations and Information Management Applications Gregor Erbach Saarland University Saarbrücken.
Linguistic Resources for the 2013 TAC KBP Entity Linking Evaluation Joe Ellis (presenter), Justin Mott, Xuansong Li, Jeremy Getman, Jonathan Wright, Stephanie.
LANGUAGE MODELS FOR RELEVANCE FEEDBACK Lee Won Hee.
PSEUDO-RELEVANCE FEEDBACK FOR MULTIMEDIA RETRIEVAL Seo Seok Jun.
A Scalable Machine Learning Approach for Semi-Structured Named Entity Recognition Utku Irmak(Yahoo! Labs) Reiner Kraft(Yahoo! Inc.) WWW 2010(Information.
PRIS at Slot Filling in KBP 2012: An Enhanced Adaboost Pattern-Matching System Yan Li Beijing University of Posts and Telecommunications
Ang Sun Director of Research, Principal Scientist, inome
Chapter 23: Probabilistic Language Models April 13, 2004.
Maximum Entropy Models and Feature Engineering CSCI-GA.2590 – Lecture 6B Ralph Grishman NYU.
Authors: Marius Pasca and Benjamin Van Durme Presented by Bonan Min Weakly-Supervised Acquisition of Open- Domain Classes and Class Attributes from Web.
ASSESSING LEARNING ALGORITHMS Yılmaz KILIÇASLAN. Assessing the performance of the learning algorithm A learning algorithm is good if it produces hypotheses.
Bag-of-Words Methods for Text Mining CSCI-GA.2590 – Lecture 2A Ralph Grishman NYU.
Creating Subjective and Objective Sentence Classifier from Unannotated Texts Janyce Wiebe and Ellen Riloff Department of Computer Science University of.
Total Recall: Automatic Query Expansion with a Generative Feature Model for Object Retrieval O. Chum, et al. Presented by Brandon Smith Computer Vision.
Machine Learning Tutorial-2. Recall, Precision, F-measure, Accuracy Ch. 5.
Relational Duality: Unsupervised Extraction of Semantic Relations between Entities on the Web Danushka Bollegala Yutaka Matsuo Mitsuru Ishizuka International.
Comparing Document Segmentation for Passage Retrieval in Question Answering Jorg Tiedemann University of Groningen presented by: Moy’awiah Al-Shannaq
Mining Dependency Relations for Query Expansion in Passage Retrieval Renxu Sun, Chai-Huat Ong, Tat-Seng Chua National University of Singapore SIGIR2006.
KNN & Naïve Bayes Hongning Wang Today’s lecture Instance-based classifiers – k nearest neighbors – Non-parametric learning algorithm Model-based.
Class Imbalance in Text Classification
Discovering Relations among Named Entities from Large Corpora Takaaki Hasegawa *, Satoshi Sekine 1, Ralph Grishman 1 ACL 2004 * Cyberspace Laboratories.
Exploiting Named Entity Taggers in a Second Language Thamar Solorio Computer Science Department National Institute of Astrophysics, Optics and Electronics.
Statistical Machine Translation Part II: Word Alignments and EM Alex Fraser Institute for Natural Language Processing University of Stuttgart
Identifying “Best Bet” Web Search Results by Mining Past User Behavior Author: Eugene Agichtein, Zijian Zheng (Microsoft Research) Source: KDD2006 Reporter:
Cold-Start KBP Something from Nothing Sean Monahan, Dean Carpenter Language Computer.
Matching References to Headers in PDF Papers Tan Yee Fan 2007 December 19 WING Group Meeting.
Integrating linguistic knowledge in passage retrieval for question answering J¨org Tiedemann Alfa Informatica, University of Groningen HLT/EMNLP 2005.
Learning to Extract CSCI-GA.2590 Ralph Grishman NYU.
KNN & Naïve Bayes Hongning Wang
Relation Extraction CSCI-GA.2591
Lei Sha, Jing Liu, Chin-Yew Lin, Sujian Li, Baobao Chang, Zhifang Sui
Introduction Task: extracting relational facts from text
Presentation transcript:

Ang Sun Ralph Grishman Wei Xu Bonan Min November 15, 2011 TAC 2011 Workshop Gaithersburg, Maryland USA

 Overview of 2011 System  Baseline: 2010 System  Distant Learning  Passage Retrieval (QA)  Result Analysis  Distant Learning for Slot Filling  Class Label Refinement  Undersampling the Majority Classes  Contribution of Coreference  Experimental Results

 Baseline: 2010 System (three basic components) 1) Document Retrieval  Use Lucene to retrieve a maximum of 300 documents  Query: the query name and some minor name variants 2) Answer Extraction  Begins with text analysis: POS tagging, chunking, name tagging, time expression tagging, and coreference  Coreference is used to fill alternate_names slots  Other slots are filled using patterns (hand-coded and created semi-automatically using bootstrapping) 3) Merging  Combines answers from different documents and passages, and from different answer extraction procedures

 Distant Learning (the general algorithm)  Map relations in knowledge bases to KBP slots  Search corpora for sentences that contain name pairs  Generate positive and negative training examples  Train classifiers using generated examples  Fill slots using trained classifiers

 Distant Learning (some details of the NYU system)  Map 4.1M Freebase relation instances to 28 slots  Given a pair of names occurring together in a sentence in the KBP corpus, treat it as a ▪ positive example if it is a Freebase relation instance ▪ negative example if is not a Freebase instance but is an instance for some j'  j.  Train classifiers using MaxEnt  Fill slots using trained classifiers, in parallel with other components of NYU system

 Passage Retrieval (QA)  For each slot, a set of index terms is generated using distant supervision (using Freebase)  Terms are used to retrieve and rank passages for a specific slot  An answer is then selected based on name type and distance from the query name  Due to limitations of time, this procedure was only implemented for a few slots and was used as a fall-back strategy, if the other answer extraction components did not find any slot fill.

 Result Analysis (NYU2 R/P/F 25.5/35.0/29.5)

 Problems  Problem 1: Class labels are noisy ▪ Many False Positives because name pairs are often connected by non-relational contexts FALSE POSITIVES FALSE POSITIVES

 Problems  Problem 1: Class labels are noisy ▪ Many False Negatives because of incompleteness of current knowledge bases

 Problems  Problem 2: Class distribution is extremely unbalanced ▪ Treat as negative if is NOT a Freebase relation instance ▪ Positive VS negative: 1:37 ▪ Treat as negative if is NOT a Freebase instance but is an instance for some j'  j AND is separated by no more than 12 tokens ▪ Positive VS negative: 1:13 ▪ Trained classifiers will have low recall, biased towards negative

 Problems  Problem 3: training ignores co-reference info ▪ Training relies on full name match between Freebase and text ▪ But partial names (Bill, Mr. Gates …) occur often in text ▪ Use co-reference during training? ▪ Co-reference module itself might be inaccurate and adds noise to training ▪ But can it help during testing?

 Solutions to Problems  Problem 1: Class labels are noisy ▪ Refine class labels to reduce noise  Problem 2: Class distribution is extremely unbalanced ▪ Undersample the majority classes  Problem 3: training ignores co-reference info ▪ Incorporate coreference during testing

 The refinement algorithm I. Represent a training instance by its dependency pattern, the shortest path connecting the two names in the dependency tree representation of the sentence II. Estimate precision of the pattern Precision of a pattern p for the class C i is defined as the number of occurrences of p in the class C i divided by the number of occurrences of p in any of the classes C j III. Assign the instance the class that its dependency pattern is most precise about

 The refinement algorithm (cont)  Examples Jon Corzine, the former chairman and CEO of Goldman Sachs William S. Paley, chairman of CBS … … Example Sentence PERSON: Employee_of ORG: Founded_by Class prec(appos chairman prep_of, PERSON:Employee_of) = prec(appos chairman prep_of, ORG:Founded_by) = PERSON: Employee_of PERSON: Employee_of appos chairman prep_of

 Effort 1: multiple n-way instead of single n-way classification  single n-way: an n-way classifier for all classes ▪ Biased towards majority classes  multiple n-way: an n-way classifier for each pair of name types ▪ A classifier for PERSON and PERSON ▪ Another one for PERSON and ORGANIZATION ▪ … …  On average (10 runs on 2011 evaluation data) ▪ single n-way: 180 fills for 8 slots ▪ multiple n-way:240 fills for 15 slots

 Effort 2:  Even with multiple n-way classification approach  OTHER (not a defined KBP slot) is still the majority class for each such n-way classifier  Downsize OTHER by randomly selecting a subset of them

 No use of co-reference during training  Run Jet (NYU IE toolkit) to get co-referred names of the query  Use these names when filling slots for the query  Co-reference is beneficial to our official system  P/R/F of the distant filler itself ▪ With co-reference: 36.4/11.4/17.4 ▪ Without co-reference: 28.8/10.0/14.3

Undersampling Ratio: ratio between negatives and positives  Multiple n-way outperformed single n-way  Models with refinement:  higher performance  curves are much flatter  less sensitive to undersampling ratio  more robust to noise

 Models with refinement have better P, R  Multiple n-way outperforms single n-way mainly through improved recall

Thanks!

?