AUTOMATIC GLOSS FINDING for a Knowledge Base using Ontological Constraints Bhavana Dalvi (PhD Student, LTI) Work done with: Prof. William Cohen, CMU Prof. Einat Mikov, University of Haifa Prof. Partha Talukdar, IISC Bangalore 1

Motivation 2

Need for gloss finding  KBs are useful for many NLP tasks: E.g. Question answering  Lot of research in fact extraction to populate KBs  Glosses can help further in applications like - Word/Entity sense disambiguation - Information retrieval  Automatically constructed KBs lack glosses e.g. NELL, YAGO 3

Example: Gloss finding Class constraints:  Inclusion: Every entity that is of type “Fruit” is also of type “Food”.  Mutual Exclusion: If an entity is of type “Food” then it cannot be of type “Organization” 4

Example: Gloss finding 5

6

7 Knowledge Bases: NELL / Freebase / YAGO Candidate glosses: DBPedia abstracts/ Wiktionary definitions

Gloss Finding 8

Problem Definition InputExample KB classesFood, Fruits, Company … Ontological constraints: Subset, Mutex Entities ‘E’ belonging to KB categoriesBanana, Microsoft Lexical strings ‘L’ that refer to entities ‘E’E.g. ‘MS’, ‘microsoft inc’ Candidate glossesE.g. G3: Apple, formerly Apple Computer Inc., is an American multinational corporation headquartered in Cupertino … Output: Matching candidate glosses to entities in the KB E.g. (Apple, G3)  Company:Apple 9

Can we use existing techniques?  Problem: Match potential glosses to appropriate entities in the KB.  Entity linking: Assume existence of glosses on KB side  Input KB does not have glosses  Chicken & egg problem  Ontology alignment: Both ends being matched are structured databases  Asymmetric problem: Structured KB on one side, without glosses Candidate glosses contain text but no structure 10

Proposed Gloss Finding Procedure  Decide head-NP for a gloss: NP being defined G3: Apple, formerly Apple Computer Inc., is an American multinational corporation headquartered in Cupertino …  Select candidate glosses for which string match (head-NP, KB entity)  For each gloss  a set of candidate KB entities (Apple, G3)  (Fruit:Apple, Company:Apple)  Classify the head-NP into KB classes using ontological constraints (Apple, G3)  Company  Choose the KB entity match based on chosen KB category. (Apple, G3)  Company:Apple 11

Building Classifiers 12

Training classifiers for KB Categories Train: Unambiguous glosses Test: Ambiguous glosses 13

Assumptions  If a gloss has only one candidate entity matching in a KB, then it is correct  i.e. we assume that KB is always correct and complete in terms of senses.  Assumption holds for 81% for NELL dataset  Given the category, a mention is unambiguous [Suchanek WWW’07, Nakashole ACL’13]  i.e. we can differentiate between entities of different category but not within a category. 14

Methods  Baselines  SVM Learning Train binary classifiers using unambiguous glosses Predict categories for ambiguous glosses  Label propagation PIDGIN [Wijaya et al. CIKM’13]: Graph-based label propagation method.  GLOFIN: semi-supervised EM + use of ontological constraints. 15

Proposed Method: GLOFIN Initialize model with few seeds per class Iterate till convergence (Data likelihood) E step: Predict labels for unlabeled points For each unlabeled datapoint  Find P(Class | datapoint) for all classes  Assign a consistent bit vector of labels in accordance with ontological constraints M step: Recompute model parameters using seeds + predicted labels for unlabeled points 16

Proposed Method: GLOFIN Initialize model with few seeds per class Iterate till convergence (Data likelihood) E step: Predict labels for unlabeled points For each unlabeled datapoint  Find P(Class | datapoint) for all classes  Assign a consistent bit vector of labels in accordance with ontological constraints M step: Recompute model parameters using seeds + predicted labels for unlabeled points 17

Estimating class parameters and assignment probabilities  Naïve Bayes Independent multinomial distributions per word  K-Means Cosine similarity between centroid and datapoint  von-Mises Fisher Data distributed on a unit hypersphere 18

Proposed Method: GLOFIN Initialize model with few seeds per class Iterate till convergence (Data likelihood) E step: Predict labels for unlabeled points For each unlabeled datapoint  Find P(Class | datapoint) for all classes  Assign a consistent bit vector of labels in accordance with ontological constraints M step: Recompute model parameters using seeds + predicted labels for unlabeled points 19

Mixed Integer Linear Program Input: P(C j | X i ), Class constraints: Subset, Mutex Output: Consistent bit vector y ji for X i Max { likelihood of assignment – constraint violation penalty } 20

Proposed Method: GLOFIN Initialize model with few seeds per class Iterate till convergence (Data likelihood) E step: Predict labels for unlabeled points For each unlabeled datapoint  Find P(Class | datapoint) for all classes  Assign a consistent bit vector of labels in accordance with ontological constraints M step: Recompute model parameters using seeds + predicted labels for unlabeled points 21

Experiments 22

Candidate glosses  DBPedia is a database derived from Wikipedia  We use short abstracts (definitions upto 500 characters, from Wikipedia page)  E.g. McGill University is a research university located in Montreal Quebec Canada Founded in 1821 during the British colonial era the university bears the name of James McGill a prominent Montreal merchant from Glasgow Scotland and alumnus of Glasgow University whose bequest formed the beginning of the university. 23

Knowledge bases 24

GLOFIN vs. SVM & Label propagation Freebase Dataset: Performance on ambiguous glosses 25

GLOFIN vs. SVM & Label propagation NELL Dataset: Performance on ambiguous glosses 26

Are the datasets close to real world?  Large fraction of data used for training 80% of NELL 90% of Freebase  In real world scenarios, amount of training data might be a small fraction of the dataset.  We simulate this by using 10% of unambiguous glosses for training 27

Small amount of training data Freebase Dataset: Performance on ambiguous glosses 28

Small amount of training data NELL Dataset: Performance on ambiguous glosses 29

Compare variants of GLOFIN Freebase Dataset 30

Compare variants of GLOFIN NELL Dataset 31

Some more experiments … 32  Evaluating quality of automatically acquired seeds  Manually creating gold standard for NELL dataset  Different ways of scaling GLOFIN  NELL to Freebase mappings via common glosses

And Future Work ….. Conclusions 33

Conclusions  Completely unsupervised method for gloss finding - using unambiguous matches as training data - hierarchical classification instead of entity linking  Our proposed method GLOFIN: GLOFIN ≥ Label Propagation ≥ SVM  Variants of Hierarchical GLOFIN Naïve Bayes ≥ K-Means, von-Mises Fisher  Ontological constraints help for all GLOFIN variants Hierarchical GLOFIN ≥ Flat GLOFIN  In future, we will like to add new entities to the KB. 34

head-NPGloss Candidate NELL entities Entity selected by GLOFIN McGill_UniversityMcGill University is a research university located in Montreal Quebec Canada Founded in 1821 during the British colonial era the university bears the name of James McGill a prominent … University:E, Sports_team:E University:E Kingston_upon_ Hull Kingston upon Hull frequently referred to as Hull is a city and unitary authority area in the ceremonial county of the East Riding of Yorkshire England It stands on the River Hull at its junction with … City:E, Visual_Artist:E City:E Robert_SoutheyRobert Southey was an English poet of the Romantic school one of the so called Lake Poets and Poet Laureate for 30 years from 1813 to his death in 1843 Although his fame has been long eclipsed by that … Person_Europe:E, Person_Africa:E, Politician_USA:E Person_Europe:E 35

Thank You Questions? 36

Extra Slides 37

Comparing of GLOFIN Approximations 38

Eval: quality of seeds for NELL KB  Noisy seeds: Only 81% leaf category assignments are correct  Hierarchical labeling can help: 94% higher level category labels are correct 39

Creating gold standard for NELL  Gold standard for evaluation on ambiguous glosses  For most glosses, precise category is part of NELL 40

NELL – Freebase mappings via common glosses 41

Pros and Cons of GLOFIN  Generative EM framework that can build on SSL methods: NBayes, K-Means, VMF  Can label unseen datapoints once models are learnt.  Assumption: Input KB is complete and accurate.  All experiments are done in transductive setting: need to extend for missing entities and categories in the KB. AdvantagesLimitations 42

Future work …  Adding new entities to existing KB categories  KBs are usually incomplete w.r.t coverage of entities.  GLOFIN: Classifies mentions into categories  Introducing new clusters of entities: missing categories in the KB  Extensions similar to Exploratory EM [Dalvi et al. ECML’13]  New categories: entities belonging to them, along with glosses for those entities 43