Evaluating Semantic Metadata without the Presence of a Gold Standard Yuangui Lei, Andriy Nikolov, Victoria Uren, Enrico Motta Knowledge Media Institute, The Open University

Focuses A quality model which characterizes quality problems in semantic metadata An automatic detection algorithm Experiments

Ontology Metadata Data

Semantic Metadata Generation Semantic Metadata Acquisition Semantic Metadata Repositories

Semantic Metadata Generation Semantic Metadata Acquisition Semantic Metadata Repositories A number of problems can happen that decrease the quality of metadata

Quality Evaluation Metadata providers: ensuring high quality Users: facilitate assessing the trustworthiness Applications: filtering out poor quality data

Our Quality Evaluation Framework A quality model Assessment metrics An automatic evaluation algorithm

The Quality Model Real World Semantic Metadata OntologiesData Sources Modelling Instantiating Annotating Representing Describing

Quality Problems (a) Incomplete Annotation Data ObjectsSemantic Entities

Quality Problems (a) Incomplete Annotation (b) Duplicate Annotation

Quality Problems (a) Incomplete Annotation (b) Duplicate Annotation (c) Ambiguous Annotation

Quality Problems (a) Incomplete Annotation (b) Duplicate Annotation (c) Ambiguous Annotation (d) Spurious Annotation

Quality Problems (a) Incomplete Annotation (b) Duplicate Annotation (c) Ambiguous Annotation (d) Spurious Annotation (e) Inaccurate Annotation

Quality Problems (a) Incomplete Annotation (b) Duplicate Annotation (c) Ambiguous Annotation (d) Spurious Annotation (e) Inaccurate Annotation Semantic metadata I1 I2 I3 R1 R2 Class C1 C2 C3 I4 R2 (f) Inconsistent Annotation

Current Support for Evaluation Gold standard based: –Examples: Gate[1], LA[2], BDM[3] Feature: assessing the performance of information extraction techniques used. Not suitable for evaluating semantic metadata –Gold standard annotations are often not available

The Semantic Metadata Acquisition Scenario KMi News Stories Information Extraction Engine (ESpotter) Semantic Data Transformation Engine Departmental Databases Raw Metadata High Quality Metadata Evaluation Evaluation needs to take place dynamically whenever a new entry is generated. In such context, gold standard is NOT available.

Our Approach Using available knowledge instead of asking for gold standard annotations –Knowledge sources specific for the domain: Domain ontologies, data repositories, domain specific lexicons –Knowledge available at background Semantic Web, Web, and general lexicon resources Advantages: –Making possible for automatic operation –Making possible for large scale data evaluation

Using Domain Knowledge 1. Domain Ontologies Constraints and restrictions Inconsistent Problems Example: one person classified as both KMi-Member and None-KMi-Member when they are disjoint classes.

Using Domain Knowledge 1. Domain Ontologies Constraints and restrictions Inconsistent Annotations 2. Domain Lexicons Lexicon – instance mappings Duplicate Annotations Example: when OU and Open-University both appear as values of the same property of the same instance

Using Domain Knowledge 1. Domain Ontologies Constraints and restrictions Inconsistent Annotations 2. Domain Lexicons Lexicon – instance mappings Duplicate Annotations 3. Domain Data Repositories Ambiguous Annotations Inaccurate Annotations

When nothing can be found in the domain knowledge, the data can be: –Correct but outside the domain (e.g., IBM in the KMi domain) –Inaccurate annotation: mis-classification (e.g., Sun Micro-systems as a person) –Spurious (e.g., workshop chair as an organization) Background knowledge is then used to further investigate the problems

Semantic Web Investigating the Semantic Web Classes Similar? Found matches No Yes Examining the Web No Inaccurate Annotations Watson WordNet Yes Adding data to the repositories

Pankow Web Examining the Web Similar? Has classification? No Yes No Inaccurate Annotations Spurious Annotations WordNet

The Overall Picture Web Semantic Web Background Knowledge Domain Knowledge Metadata Evaluation Results Ontologies Lexical Resources WordNet Web PANKOWWATSON Semantic Web SemSearch Step1: Using domain knowledge Step2: Using background knowledge Evaluation Engine Pellet + Reiter

(a) Incomplete Annotation (b) Duplicate Annotation (c) Ambiguous Annotation (d) Spurious Annotation (e) Inaccurate Annotation Semantic metadata I1 I2 I3 R1 R2 Class C1 C2 C3 I4 R2 (f) Inconsistent Annotation Addressed Quality Problems

Experiments Data settings: gathered in our previous work [4] in KMi semantic web portal –Randomly chose 36 news stories from the KMi news archive –Collected a metadata set by using ASDI –Constructed a gold standard annotation Method: –A gold standard based evaluation as a comparison base line –Evaluating the data set using domain knowledge only –Evaluating the data set using both domain knowledge and background knowledge

A number of entities are not contained in the problem domain

Background knowledge is useful in data evaluation

Discussion The performance of such an approach largely depends on: –A good domain specific knowledge source –A good publicity of the entities that are contained in the data set, otherwise there would be lots of false alarms.

References 1.H. Cunningham, D. Maynard, K. Bontcheva, and V. Tablan. GATE: A Framework and Graphical Development Environment for Robust NLP Tools and Applications. In Proceedings of the 40 th Anniversary Meeting of the Association for Computational Linguistics (ACL02), P. Cimiano, S. Staab, and J. Tane. Acquisition of Taxonomies from Text: FCA meets NLP. In Proceedings of the ECML/PKDD Workshop on Adaptive Text Extraction and Mining, pages 10 – 17, D. Maynard, W. Peters, and Y. Li. Metrics for Evaluation of Ontology-based Information Extraction. In Proceedings of the 4th International Workshop on Evaluation of Ontologies on the Web, Edinburgh, UK, May Y. Lei, M. Sabou, V. Lopez, J. Zhu, V. S. Uren, and E. Motta. An Infrastructure for Acquiring High Quality Semantic Metadata. In Proceedings of the 3rd European Semantic Web Conference, 2006.