Experiences in Evaluation and Selection of Ontologies Bruno Grilo INESC-ID H. Sofia Pinto IST/INESC-ID

Experiences in Evaluation and Selection of Ontologies Overview Introduction Methodology Setting Process Results Conclusions Overview Introduction Evaluation Methodologies Setting of the Experiences Process Results of the Evaluation Conclusions and Future Work

Experiences in Evaluation and Selection of Ontologies Overview Introduction Methodology Setting Process Results Conclusions Introduction Ontology Engineering Still remains an art, methodologies needed: For construction For evaluation Why Evaluate? Assure quality Currently 2 main methodologies in literature Gómez-Pérez OntoClean

Experiences in Evaluation and Selection of Ontologies Overview Introduction Methodology Setting Process Results Conclusions Gómez-Pérez's Methodology Verify Completeness – complete if everything it should say (and (or - Is explicitly represented - May be inferred from definitions and axioms) - Each individual definition is complete) Consistency Not possible to infer contradictions Conciseness There are no redundancies Expandability and Robustness

Experiences in Evaluation and Selection of Ontologies Overview Introduction Methodology Setting Process Results Conclusions OntoClean Methodology Meta-properties Based on philosophical notions Used to classify: Properties Classes Relations Impose constraints to the taxonomy Meta-properties are Identity – Identifying individual entities as being the same Unity – Identifying if an entity is a whole Rigidity – Identifying if a property is essential to its instances Dependency

Experiences in Evaluation and Selection of Ontologies Overview Introduction Methodology Setting Process Results Conclusions Problems Gómez-Pérez's Methodology Vague criteria Hard for newcomers to understand how and what to do OntoClean Methodology Philosofical notions hard to understand Hard to classify concepts according to the meta-properties Result Existing methodologies are seldom used Evaluation activity Not performed most of the times

Experiences in Evaluation and Selection of Ontologies Setting of the Experiences 2 Sets of domain Ontologies Central Dogma of Biology 111 Ontologies Stars Life Cycle 25 Ontologies Golden Standards 1 for each Set Goals Select some ontologies of each set In-depth evaluate the selected ontologies Overview Introduction Methodology Setting Process Results Conclusions

Experiences in Evaluation and Selection of Ontologies Overview Introduction Methodology Setting Process Results Conclusions Process Knowledge Acquisition In-Depth Evaluation of the Golden Standard Pre-evaluation and Selection Of Ontologies In-Depth Evaluation of Selected Ontologies

Experiences in Evaluation and Selection of Ontologies Overview Introduction Methodology Setting Process Results Conclusions Selection of the Ontologies to Evaluate Analysis of each ontology Given a grade starting at 20 Completeness Errors: -1 (max -3) Wrong usage of is-a relation: -1 for each different type Modelling Errors: -2 Clarity Errors: -1 Abstraction Errors: -1 (max -3) Concision Errors: -1 Consistency Errors: -10 Selection of ontologies according to grade: Pick ontologies with grade > 13 Knowledge Acquisition In-Depth Evaluation Of the Golden Standard Pre-Evaluation and Selection Of Ontologies In-Depth Evaluation of Selected Ontologies

Experiences in Evaluation and Selection of Ontologies Gómez-Pérez's Methodology Analysis of: Completeness Consistency Conciseness Human Centered Criteria Clarity Abstraction OntoClean Methodology Meta-properties: Identity Unity Rigidity Constraint Satisfaction Overview Introduction Methodology Setting Process Results Conclusions In-Depth Evaluation - Process OntoClean Preparation Phase Discard properties and relations Gómez-Pérez Classification According to Meta-Properties Verify Constraints Correct Taxonomy Reinsert Properties and Relations Analysis of Completeness Consistency and Conciseness Analysis of Clarity And Abstraction Correct Ontology Knowledge Acquisition In-Depth Evaluation Of the Golden Standard Pre-Evaluation and Selection Of Ontologies In-Depth Evaluation of Selected Ontologies Human Centered Criteria

Experiences in Evaluation and Selection of Ontologies Overview Introduction Methodology Setting Process Results Conclusions Abstraction and Clarity Abstraction Important to ontology–application connection Evaluated through Identity Criteria Find equal criteria in disjoint classes Balance Comparing each level with other ontologies of the same domain Clarity Important for the evaluator, user, maintenance team Evaluated through Comparison of the name with documented definition Search knowledge sources when there is no documentation

Experiences in Evaluation and Selection of Ontologies Star's Life Cycle Overview Introduction Methodology Setting Process Results Conclusions Evaluation Results (1) Error Tables Central Dogma of Biology 98.20%Total (%) 109Total Complet % 90 Modelling 6.31% 33.33%78.38%4.50%9.01%45.95%21.62% Is-a disj.Is-a POIs-a Inst.Concise.Consist.AbstractionClarity 87.50%Total (%) 21Total Complet % 22 Modelling 41.67%45.83%79.17%12.50%29.17%4.17%37.50% Is-a disj.Is-a POIs-a Inst.Concise.Consist.AbstractionClarity

Experiences in Evaluation and Selection of Ontologies Evaluation Results(2) Star's Life Cycle 1st Ontologies 5 Last Ont. 41st Ont. 2 Last Ont. 1.5 Steps 1 to 4 20 Step Steps 5 to 7 4 Time (h) 5 5 Golden Standard (150 cl./50 rel.) Pre-evaluation and Selection (45 cl./25 rel.) In-Depth evaluation (9 ontologies) Overview Introduction Methodology Setting Process Results Conclusions Total Time: 24h... Total Time: 40h Total Time: 40h...

Experiences in Evaluation and Selection of Ontologies Overview Introduction Methodology Setting Process Results Conclusions Evaluation Results (3) Central Dogma of Biology 1st Ontologies 4 Last Ont st Ont. 1.5 Last Ont Steps 1 to 4 3 Step 5 to Golden Standard (360 cl./40 rel.) Pre-evaluation and Selectio (40 cl./25 rel.) In-Depth evaluation (33 ontologies) Time (h)... Total Time: 13h Total Time: 120h 4... Total Time: 100h

Experiences in Evaluation and Selection of Ontologies Overview Introduction Methodology Setting Process Results Conclusions Conclusions - Findings Error is-a as instantiation In many cases ambiguous Classification according to meta-properties Varies over time for the same evaluator Abstraction Eases the usage of the ontology in real applications Should be possible to evaluate

Experiences in Evaluation and Selection of Ontologies Overview Introduction Methodology Setting Process Results Conclusions Conclusions – Recommendations Attach definitions to concepts explicitly When concept is created Needed for evaluation Difficult to make them at evaluation time Might allow to avoid some modelling errors and contradictions Names Clear and intuitive (Legibility is very important)

Experiences in Evaluation and Selection of Ontologies Overview Introduction Methodology Setting Process Results Conclusions Conclusion and Future Work Ontology Construction Process More than evaluation parallel to the process Explicit evaluation phase between phases of the process Performed by evaluation team unbiased evaluation Any change to ontology performed by both teams Process Automatization Discard and Constraint steps are automatizable Classification, Taxonomy, Reinsertion and Human Centered Analysis steps semi-automatizable Analysis semi-automatizable for Completeness Conciseness Team evaluation / different evaluators