1. CLASSIFYING ENTITIES INTO AN INCOMPLETE ONTOLOGY Bhavana Dalvi, William W. Cohen, Jamie Callan School of Computer Science, Carnegie Mellon University

2. Motivation  Existing Techniques  Semi-supervised Hierarchical Classification: Carlson WSDM’10  Extending knowledge bases: Finding new relations or attributes of existing concepts Mohamed et al. EMNLP’11  Unsupervised ontology discovery: Adams et al. NIPS’10, Blei et al. JACM’10, Reisinger et al. ACL’09  Evolving Web-scale datasets  Billions of entities and hundreds of thousands of concepts  Difficult to create a complete ontology  Hierarchical classification of entities into incomplete ontologies is needed

3. Contributions  Hierarchical Exploratory EM  Adds new instances to the existing classes  Discovers new classes and adds them at appropriate places in the ontology  Class constraints:  Inclusion: Every entity that is “Mammal” is also an “Animal”  Mutual Exclusion: If an entity is “Electronic Device” then its not “Mammal”

4. Problem Definition

5. Review: Exploratory EM [Dalvi et al. ECML 2013] Initialize model with few seeds per class Iterate till convergence (Data likelihood and # classes) E step: Predict labels for unlabeled points If P(Cj | Xi) is nearly-uniform for a data-point Xi, j=1 to k  Create a new class C k+1, assign Xi to it M step: Recompute model parameters using seeds + predicted labels for unlabeled points  Number of classes might increase in each iteration Check if model selection criterion is satisfied If not, revert to model in Iteration `t-1’ Classification/clustering KMeans, NBayes, VMF … Max/Min ratio JS Divergence AIC, BIC, AICc …

6. Hierarchical Exploratory EM

7. Divide-And-Conquer Exploratory EM Mutual ExcIusion Root Food Location Country State Vegetable Condiment Inclusion E.g. Spinach, Potato, Pepper… Level 1 Level 2 Level 3 Assumptions:  Classes are arranged in a tree- structured hierarchy.  Classes at any level of the hierarchy are mutually exclusive.

8. Divide-And-Conquer Exploratory EM Root Food Location Country State Vegetable Condiment 1.0 California

9. Divide-And-Conquer Exploratory EM Root Food Location Country State Vegetable Condiment 1.0 California 0.9 0.1

10. Divide-And-Conquer Exploratory EM Root Food Location Country State Vegetable Condiment 1.0 California 0.8 0.2 0.9 0.1 0 1 0 0 1 1 0

11. Divide-And-Conquer Exploratory EM Root Food Location Country State Vegetable Condiment 1.0 Coke

12. Divide-And-Conquer Exploratory EM Root Food Location Country State Vegetable Condiment 1.0 Coke 0.1 0.9

13. Divide-And-Conquer Exploratory EM Root Food Location Country State Vegetable Condiment 1.0 Coke 0.1 0.9 0.550.45

14. Divide-And-Conquer Exploratory EM Root Food Location Country State Vegetable Condiment 1.0 Coke 0.1 0.9 0.550.45 C8 Coke 1 0 0 0 1 0 0 1

15. Divide-And-Conquer Exploratory EM Root Food Location Country State Vegetable Condiment 1.0 Coke 0.1 0.9 0.550.45 1 0 0 0 1 0 0 1 Coke

16. Divide-And-Conquer Exploratory EM Root Food Location Country State Vegetable Condiment 1.0 Cat C8 C9 0.450.55 Cat 0 0 0 0 0 0 0 1 1

17. What are we trying to optimize? Objective Function : Maximize { Log Data Likelihood – Model Penalty } m: #clusters, Params{C1… Cm} subject to Class constraints: Z m

18. Datasets Ontology 1 Ontology 2 Dataset#Classes#Levels#NELL entities #Contexts DS-11132.5K3.4M DS-239412.9K6.7M Clueweb09 Corpus + Subsets of NELL

19. Results Dataset#Train /Test Points DS-1335/ 2.2K DS-21.5K/ 11.4K

20. Results Dataset#Train /Test Points Level#Seed/ #Ideal Classes DS-1335/ 2.2K 22/3 34/7 DS-21.5K/ 11.4K 23.9/4 39.4/24 42.4/10

21. Results Dataset#Train /Test Points Level#Seed/ #Ideal Classes Macro-averaged Seed Class F1 FLAT SemisupEMExploratoryEM DS-1335/ 2.2K 22/343.278.7 * 34/734.442.6 * DS-21.5K/ 11.4K 23.9/464.353.40 39.4/2431.333.7 * 42.4/1027.538.9 *

22. Results Dataset#Train /Test Points Level#Seed/ #Ideal Classes Macro-averaged Seed Class F1 FLATDAC SemisupEMExploratoryEMSemisupEMExploratoryEM DS-1335/ 2.2K 22/343.278.7 *69.577.2 * 34/734.442.6 *31.344.4 * DS-21.5K/ 11.4K 23.9/464.353.4065.468.9 * 39.4/2431.333.7 *34.941.7 * 42.4/1027.538.9 *43.242.40

23. Conclusions

24. Thank You Questions?

25. Extra Slides

26. Class Creation Criterion

27. Model Selection Extended Akaike Information Criterion AICc(g) = -2*L(g) + 2*v + 2*v*(v+1)/(n – v -1) Here g: model being evaluated, L(g): log-likelihood of data given g, v: number of free parameters of the model, n: number of data-points. 