Linguistic Resources for the 2013 TAC KBP Entity Linking Evaluation Joe Ellis (presenter), Justin Mott, Xuansong Li, Jeremy Getman, Jonathan Wright, Stephanie Strassel Linguistic Data Consortium University of Pennsylvania, USA

2013 Source Corpus LanguageGenreDocuments English Newswire1,000,257 Web Text 999,999 Discussion Forums 99,063 Chinese Newswire 2,000,256 Web Text 815,886 Discussion Forums199,321 SpanishNewswire 910,734 TAC KBP Evaluation Workshop – NIST, November 18-19, 2013

Entity Linking Overview TAC KBP Evaluation Workshop – NIST, November 18-19, 2013 Stage 1: Select name strings and ref docs Stage 2: Link namestrings to KB or mark as NIL

Entity Linking Overview TAC KBP Evaluation Workshop – NIST, November 18-19, 2013 Stage 1: Select name strings and ref docs Stage 2: Link namestrings to KB or mark as NIL

Entity Linking Overview TAC KBP Evaluation Workshop – NIST, November 18-19, 2013 Stage 1: Select name strings and ref docs Stage 2: Link namestrings to KB or mark as NIL

Entity Linking Overview TAC KBP Evaluation Workshop – NIST, November 18-19, 2013 Stage 1: Select name strings and ref docs Stage 2: Link namestrings to KB or mark as NIL Stage 3: Co-reference NIL entities Wendy Wendy Gaxiola

Entity Linking Overview TAC KBP Evaluation Workshop – NIST, November 18-19, 2013 Stage 1: Select name strings and ref docs Stage 2: Link namestrings to KB or mark as NIL Stage 3: Co-reference NIL entities Wendy Wendy Gaxiola

Entity Linking – Stage 1  Run named entity taggers over source corpora Provides guided search through the corpus Thanks KBP coordinators!  Namestring Selection Confusable TAC KBP Evaluation Workshop – NIST, November 18-19, 2013

Entity Linking – Stage 1  Run named entity taggers over source corpora Provides guided search through the corpus Thanks KBP coordinators!  Namestring Selection Confusable TAC KBP Evaluation Workshop – NIST, November 18-19, 2013

Entity Linking – Stage 1  Run named entity taggers over source corpora Provides guided search through the corpus Thanks KBP coordinators!  Namestring Selection Confusable TAC KBP Evaluation Workshop – NIST, November 18-19, 2013

Entity Linking – Stage 1  Run named entity taggers over source corpora Provides guided search through the corpus Thanks KBP coordinators!  Namestring Selection Confusable Ambiguous Varied TAC KBP Evaluation Workshop – NIST, November 18-19, 2013 ?

Entity Linking – Stage 1  Run named entity taggers over source corpora Provides guided search through the corpus Thanks KBP coordinators!  Namestring Selection Confusable Ambiguous Varied TAC KBP Evaluation Workshop – NIST, November 18-19, 2013 ?

Entity Linking – Stage 1: Namestring Selection  Ratios NIL & non-NIL Entity types Genre  Measurable confusability Multiple-entity namestrings (“Smith”) Multiple-namestring entities (“Barack Obama”, “Bam-Bam”, “Bammy”) NIL singletons Cross-lingual TAC KBP Evaluation Workshop – NIST, November 18-19, 2013

Entity Linking – Stages 2 & 3: KB Linking and NIL Coref  KB Linking Review ref document and search KB for matching node Multiple namestrings viewed together for quicker linking  NIL Coreference NIL queries (no KB match) require manual co-reference annotation Time-limited quality control pass to enhance completeness and accuracy TAC KBP Evaluation Workshop – NIST, November 18-19, 2013

Delivered 2013 Resources TAC KBP Evaluation Workshop – NIST, November 18-19, 2013 Corpus TitleTypeLDC CatalogLanguageSize TAC 2013 KBP English Entity Linking Evaluation Queries and Knowledge Base Links EvaluationLDC2013E90English 803 GPE 686 PER 701 ORG TAC 2013 KBP Chinese Entity Linking Evaluation Queries and Knowledge Base Links EvaluationLDC2013E96 Chinese English 714 GPE 706 PER 735 ORG TAC 2013 KBP Spanish Entity Linking Evaluation Queries and Knowledge Base Links EvaluationLDC2013E97 Spanish English 660 GPE 695 PER 762 ORG