A Framework for Benchmarking Entity-Annotation Systems

Slides:

Advertisements

Similar presentations

Date: 2014/05/06 Author: Michael Schuhmacher, Simon Paolo Ponzetto Source: WSDM’14 Advisor: Jia-ling Koh Speaker: Chen-Yu Huang Knowledge-based Graph Document.

Advertisements

Search in Source Code Based on Identifying Popular Fragments Eduard Kuric and Mária Bieliková Faculty of Informatics and Information.

Knowledge Base Completion via Search-Based Question Answering

Date : 2013/05/27 Author : Anish Das Sarma, Lujun Fang, Nitin Gupta, Alon Halevy, Hongrae Lee, Fei Wu, Reynold Xin, Gong Yu Source : SIGMOD’12 Speaker.

Linking Entities in #Microposts ROMIL BANSAL, SANDEEP PANEM, PRIYA RADHAKRISHNAN, MANISH GUPTA, VASUDEVA VARMA INTERNATIONAL INSTITUTE OF INFORMATION TECHNOLOGY,

Overview of the TAC2013 Knowledge Base Population Evaluation: English Slot Filling Mihai Surdeanu with a lot help from: Hoa Dang, Joe Ellis, Heng Ji, and.

Linking Named Entity in Tweets with Knowledge Base via User Interest Modeling Date : 2014/01/22 Author : Wei Shen, Jianyong Wang, Ping Luo, Min Wang Source.

Bring Order to Your Photos: Event-Driven Classification of Flickr Images Based on Social Knowledge Date: 2011/11/21 Source: Claudiu S. Firan (CIKM’10)

Leveraging Community-built Knowledge For Type Coercion In Question Answering Aditya Kalyanpur, J William Murdock, James Fan and Chris Welty Mehdi AllahyariSpring.

INTRODUCTION PROBLEM FORMULATION FRAMEWORK AND PRIVACY REQUIREMENTS FOR MRSE PRIVACY-PRESERVING AND EFFICIENT MRSE PERFORMANCE ANALYSIS RELATED WORK CONCLUSION.

Explorations in Tag Suggestion and Query Expansion Jian Wang and Brian D. Davison Lehigh University, USA SSM 2008 (Workshop on Search in Social Media)

Gimme’ The Context: Context- driven Automatic Semantic Annotation with CPANKOW Philipp Cimiano et al.

Retrieval Evaluation. Introduction Evaluation of implementations in computer science often is in terms of time and space complexity. With large document.

Large-Scale Content-Based Image Retrieval Project Presentation CMPT 880: Large Scale Multimedia Systems and Cloud Computing Under supervision of Dr. Mohamed.

Word Sense Disambiguation for Automatic Taxonomy Construction from Text-Based Web Corpora 12th International Conference on Web Information System Engineering.

C OLLECTIVE ANNOTATION OF WIKIPEDIA ENTITIES IN WEB TEXT - Presented by Avinash S Bharadwaj ( )

SEEKING STATEMENT-SUPPORTING TOP-K WITNESSES Date: 2012/03/12 Source: Steffen Metzger (CIKM’11) Speaker: Er-gang Liu Advisor: Dr. Jia-ling Koh 1.

1 Context-Aware Search Personalization with Concept Preference CIKM’11 Advisor ： Jia Ling, Koh Speaker ： SHENG HONG, CHUNG.

Leveraging Conceptual Lexicon ： Query Disambiguation using Proximity Information for Patent Retrieval Date : 2013/10/30 Author : Parvaz Mahdabi, Shima.

1 Wikification CSE 6339 (Section 002) Abhijit Tendulkar.

BACKGROUND KNOWLEDGE IN ONTOLOGY MATCHING Pavel Shvaiko joint work with Fausto Giunchiglia and Mikalai Yatskevich INFINT 2007 Bertinoro Workshop on Information.

Fine-Grained Location Extraction from Tweets with Temporal Awareness Date:2015/03/19 Author:Chenliang Li, Aixin Sun Source:SIGIR '14 Advisor:Jia-ling Koh.

Date: 2013/8/27 Author: Shinya Tanaka, Adam Jatowt, Makoto P. Kato, Katsumi Tanaka Source: WSDM’13 Advisor: Jia-ling Koh Speaker: Chen-Yu Huang Estimating.

Recommending Twitter Users to Follow Using Content and Collaborative Filtering Approaches John HannonJohn Hannon, Mike Bennett, Barry SmythBarry Smyth.

Wikipedia as Sense Inventory to Improve Diversity in Web Search Results Celina SantamariaJulio GonzaloJavier Artiles nlp.uned.es UNED,c/Juan del Rosal,

1 Using The Past To Score The Present: Extending Term Weighting Models with Revision History Analysis CIKM’10 Advisor ： Jia Ling, Koh Speaker ： SHENG HONG,

Detecting Communities Via Simultaneous Clustering of Graphs and Folksonomies Akshay Java Anupam Joshi Tim Finin University of Maryland, Baltimore County.

2015/12/121 Extracting Key Terms From Noisy and Multi-theme Documents Maria Grineva, Maxim Grinev and Dmitry Lizorkin Proceeding of the 18th International.

Named Entity Disambiguation on an Ontology Enriched by Wikipedia Hien Thanh Nguyen 1, Tru Hoang Cao 2 1 Ton Duc Thang University, Vietnam 2 Ho Chi Minh.

Ranking Related Entities Components and Analyses CIKM’10 Advisor: Jia Ling, Koh Speaker: Yu Cheng, Hsieh.

Date: 2013/10/23 Author: Salvatore Oriando, Francesco Pizzolon, Gabriele Tolomei Source: WWW’13 Advisor: Jia-ling Koh Speaker: Chen-Yu Huang SEED:A Framework.

LINDEN : Linking Named Entities with Knowledge Base via Semantic Knowledge Date : 2013/03/25 Resource : WWW 2012 Advisor : Dr. Jia-Ling Koh Speaker : Wei.

Web Search Personalization with Ontological User Profile Advisor: Dr. Jai-Ling Koh Speaker: Shun-hong Sie.

Concept-based Short Text Classification and Ranking

CONTEXTUAL SEARCH AND NAME DISAMBIGUATION IN USING GRAPHS EINAT MINKOV, WILLIAM W. COHEN, ANDREW Y. NG SIGIR’06 Date: 2008/7/17 Advisor: Dr. Koh,

Topical Clustering of Search Results Date : 2012/11/8 Resource : WSDM’12 Advisor : Dr. Jia-Ling Koh Speaker : Wei Chang 1.

Finding similar items by leveraging social tag clouds Speaker: Po-Hsien Shih Advisor: Jia-Ling Koh Source: SAC 2012’ Date: October 4, 2012.

GoRelations: an Intuitive Query System for DBPedia Lushan Han and Tim Finin 15 November 2011

LEARNING IN A PAIRWISE TERM-TERM PROXIMITY FRAMEWORK FOR INFORMATION RETRIEVAL Ronan Cummins, Colm O’Riordan (SIGIR’09) Speaker : Yi-Ling Tai Date : 2010/03/15.

2016/9/301 Exploiting Wikipedia as External Knowledge for Document Clustering Xiaohua Hu, Xiaodan Zhang, Caimei Lu, E. K. Park, and Xiaohua Zhou Proceeding.

Automatically Labeled Data Generation for Large Scale Event Extraction

Concept Grounding to Multiple Knowledge Bases via Indirect Supervision

Exploiting Wikipedia as External Knowledge for Document Clustering

Medication Information Extraction

A Brief Introduction to Distant Supervision

Insert Authors’ Headshots or Group Photo Here

DM-Group Meeting Liangzhe Chen, Nov

SOCIAL COMPUTING Homework 3 Presentation

GLOW- Global and Local Algorithms for Disambiguation to Wikipedia

On the Generative Discovery of Structured Medical Knowledge

Project 4 User and Movie 2018/11/10.

Accounting for the relative importance of objects in image retrieval

Presentation 王睿.

Liang Zheng and Yuzhong Qu

[jws13] Evaluation of instance matching tools: The experience of OAEI

Reading Report on Question Answering

Learning Literature Search Models from Citation Behavior

Intent-Aware Semantic Query Annotation

CS246: Information Retrieval

Summarization for entity annotation Contextual summary

Enriching Taxonomies With Functional Domain Knowledge

Project 3 Image Retrieval

A new era: Topic-based Annotators

Related Work in Camera Network Tracking

HeteroMed: Heterogeneous Information Network for Medical Diagnosis

Wiki3C: Exploiting Wikipedia for Context-aware Concept Categorization

Precise Condition Synthesis for Program Repair

Entity Linking Survey

Heterogeneous Graph Attention Network

Preference Based Evaluation Measures for Novelty and Diversity

Presentation transcript:

A Framework for Benchmarking Entity-Annotation Systems Source： WWW’13 Authors：Macro Cornolti, Paolo Ferragina and Massimiliano Advisor：Dr. Jia-Ling, Koh Speaker：Wei Chang

Outline Introduction Entity Annotation Systems Measurement Experiment Conclusion

Entity Annotation System Knowledge Base

Example

Entity Annotation Systems Many different entity annotation systems But how to compare these systems?

Goal Defining and implementing a framework for comparing in a complete, fair and meaningful way the entity annotation systems.

Outline Introduction Entity Annotation Systems Measurement Experiment Conclusion

Entity Annotation Problems Disambiguate to Wikipedia (D2W) Annotate to Wikipedia (A2W) Scored-annotate to Wikipedia (Sa2W) Concepts to Wikipedia (C2W) Scored concepts to Wikipedia (Sc2W) Ranked-concepts to Wikipedia (Rc2W)

Entity Annotation Problems

Entity Annotation Problems

Entity Annotation Problems

Entity Annotation Problems

Outline Introduction Entity Annotation Systems Measurement Experiment Conclusion

Correct Match President Barack Obama issues Iran ultimatum

Generalize Standard Evaluation s: solution found by tested system g: ground truth M: binary relation M which specifies the notion of correct match"

Precision, Recall & F1

Macro and Micro The macro- measures are the average of the corresponding measure over each document in the dataset D, while the micro- measures take into account all annotations together thus giving more importance to documents with more annotations.

Correct Match If the entity found by the system and the ground truth entity redirect to the same entity, then set the correctness true. The mention and the entity correct must be right(strong annotation match). The mention overlap and the entity is right(weak annotation match).

Similarity between Systems

Outline Introduction Entity Annotation Systems Measurement Experiment Conclusion

Dataset

News

Similarity between Systems

Tweet

Web

Runtime Efficiency

Outline Introduction Entity Annotation Systems Measurement Experiment Conclusion

Conclusion We designed, implemented and tested a benchmarking framework to fairly and fully compare entity-annotation systems. It is written in Java and it has been released to the public as open source code in https://github.com/marcocor/bat- framework .