1. A Panoramic Approach to Integrated Evaluation of Ontologies in the Semantic Web S. Dasgupta, D. Dinakarpandian, Y. Lee School of Computing and Engineering University of Missouri-Kansas City

2. Overview Motivation Approach - Pan-Onto-Eval 1.Triple Centricity 2.Theme Centricity 3.Structure Centricity 4.Domain Centricity Experiments Evaluation Conclusion

3. Related Work Ontology ranking by cross-references: Swoogle [3,6], OntoSelect [7] and OntoKhoj [4] Structural richness – Tartir et al [8]: distribution and generic/specific super/sub concepts# [Alani et al. 16-18]. Density measure [16], centrality measure [18]. Relational richness – Tartir et al [8] - ratio of #non-IS-A to #rels. – Sabou et al [2] - no consideration of the roles of concepts of relationships. Very limited work on Thematic richness - multiple hierarchies in a single ontology

4. NO! Actually, they are similar They live in the same house They have the same last name They have the same children …. Are they similar? you cannot judge them all by their "covers".

5. Ontology Evaluation How to evaluate ontology? –Some ontologies are strong in terms of structure while their relationships are weak. We need to evaluate ontologies considering different perspectives.

6. OntoSnap Framework Ontology Summarization Ontology Evaluation Ontology Categorization Ontology Query & Reasoning Ontology Integration OntoSnap

7. Summary - WINE Ontology http://www.w3.org/2002/03owlt/miscellaneous/consistent001 Total Number of Classes: 138 (Defined: 77, Imported: 61) Total Number of Datatype Properties: 1 Total Number of Object Properties: 16 (Defined: 13, Imported: 3) Total Number of Annotation Properties: 2 Total Number of Individuals: 206 (Defined: 161, Imported: 45

8. Summary - Wine 3 Ontology

9. Pan-Onto-Eval A comprehensive approach to evaluating an ontology by considering its structure, semantics, and domain 1.Triple Centricity: Information sources 2.Theme Centricity: Relation Classification 3.Structure Centricity: Relationship Inheritance 4.Domain Centricity

10. Information Source Triple Centricity capturing Information source isMadeFrom Subject (Domain)‏ Relation (Property)‏ Object (Range)‏

11. Theme Centricity Classification of Relations in Wine Domain compositionalFunctionalAttributiveSpatialTemporal Relation madeInYear madeFrom madeFromFruit madeFromGrape blendWith hasMaker drink cause hasFlavor hasColor hasSugar hasBody hasRegion isLocatedIn adjacentTo Comparative tasteBetter Expensive Conceptual Relations between domain and range concepts carry different semantic ‘senses’. for better understanding of the thematic categories of the ontology

12. Relationship Inheritance isMadeFrom IS-A hasColor Cirrhosis Cause hasMaker winery IS-A beverage hasSugar Beer Wine IS-A Specific Generic Structure Centricity Distribution of non-IS-A relations

13. Wine History Grape varieties Classification Vintages Testing Collecting Production Exporting countries Uses Health effects Packaging & Storage WIKIpedia Domain Centricity Semantic implication of each hierarchy is different - contributes differently to the semantics of the ontology as a whole.

14. Pan-Onto-Eval Ontology H1 O1 H2 Hierarchies ICIR DRDR DMF DMI ICIR DRDRR DMF1 DMI1 IC IR DRDRR DMF2 DMI2 H3 IC IR DRDRR DMF3 DMI3 ρ Panoramic Metrics Domain Importance Evaluation Score

15. Information Content (IC)‏ Domain Concepts Range Concepts D1 D2 R1 R2 R3 Triple: Domain-Property-Range Which information sources are important How Range concepts are associated - with which Domain concepts - through which Relation types Information Sources

16. Information Content (IC)‏ Domain Concept Range Concept Relation type1 Relation type2 Relation type7 Information Entropy is used to measure the significance of information sources the overall uncertainty of Range concept association... IS-A

17. Inheritance Richness (IR)‏ N: Number of domain concepts in H R(DCi)): Number of relations associated with the domain concept DCi S(DCi) Number of children under the domain concept DCi All Domain Concepts X For each X IR(X) = R(X)*S(X)‏ Average of IRs X Domain Concept Range Concept IS-A Non IS-A

18. Dimensional Richness (DR)‏ The dimensional coverage of relationships in a hierarchy. The richness of these relationships are measured by  selected range concepts  corresponding domain concepts {DCi, RCj DCk, RCl...}. {DCi, RCj DCk, RCl...}. {DCi, RCj DCk, RCl...}. {DCi, RCj DCk, RCl...}.

19. Relational Richness (RR)‏ The dimensional coverage of relations in a hierarchy. The richness of these relations are measured by  selected relations for categories in a hierarchy {Ri, Rj...}.{Rk, Rl...}.{Rm, Rn...}.{Ro, Rp...}.

20. Domain Importance (DMI)‏ The richness of the core domain(s) of hierarchy H k compared to other hierarchies.

21. Ontology Evaluation Score Combine the richness of hierarchies together into a single model that can effectively evaluate ontologies. K: the number of hierarchies in a given ontology

22. Experiments We analyze three related university ontologies – http://www.ksl.stanford.edu/projects/DAML/ksl-daml-desc.daml –http://www.ksl.stanford.edu/projects/DAML/ksl-daml-instances.daml –http://www.cs.umd.edu/projects/plus/DAML/onts/univ1.0.daml. Preprocessing – convert the DAML files to OWL using a mindswap converting tool – assign a type to the relations in these ontologies ‏ – generate summaries. The application is implemented in Java using the Protégé OWL 3.3 beta API.

23. H5: Document - attributive, functional and temporal H7: Organization - conceptual and attributive H6: Organism The evaluation score of the University-I (ρ) is 6.109

24. T he best hierarchy in O2 is H6 vs. O1's is H5 The evaluation score of the ontology ( ρ ) is 3.909.

25. The evaluation score of the University-III (ρ) is 4.567.

26. Comparison of the three ontologies

27. Conclusions Pan-Onto-Eval – A comprehensive approach to evaluating an ontology considering various aspects - structure, semantics, and domain. – A formal treatment of the model The experimental results demonstrate benefits of the proposed model. Overall, the model has great potential on evaluation of distributed knowledge in the Semantic Web. Limitations – Lack of rigorous evaluation by experts. – Preprocessing – summarization, relation type assignment – Verified for real applications.