BioHealth Informatics Group QA of Ontologies OWL Tutorial December 6 th 2005

BioHealth Informatics Group The GIGO Problem Logic may be necessary But it is not sufficient Can not validate all possible inferences in advance Instead must prove: Reasoner is sound Axioms are correct Induction does the rest

BioHealth Informatics Group Barriers to Ontology QA ►Absolute Measures ►No ‘Gold Standard’ ►Mutual cross validation only as good as the parts ►Manual checking error-prone ►and can’t measure HOW error prone because of (1) ►Comparitive Measures ►Better than worse does not imply good ►Relative Measures ►Provides unequivocal evidence of improvement ►But not of proximity to goal ►Falling error detection rate does not imply none exist

BioHealth Informatics Group Types of Axiom Quality Philosophical Rigour Ontological commitment Content correctness Fitness for purpose

BioHealth Informatics Group Philosophical Rigour ►2500 years of research ►Theories of time, mereonomy, containment ►Often FOL, so not computable ►Similar upper level ontologies ►DOLCE, BFO ►But not 100% agreement: Realist vs Cognitivist

BioHealth Informatics Group Ontological commitment ►Formally specified semantic equivalence ►Logical transformation to canonical form ►Semantically equivalent but no logical transform ►‘Fixation of femur by means of inserting pins’ ►‘Insertion of pins to fixate the femur’ ►Metamodel rules/commitments ►Arbitrary choice of preferred form ►Conventions to be applied throughout ontology ►And by all applications that use it

BioHealth Informatics Group Content correctness ►Metadata Provenance, lexical annotations etc ►Truth ►‘Structure of labial vein’ is-a ‘Superficial vein of face’ ►Completeness – ambiguity and omission ►Thymus secretes Thymosin; Thymosin is-a Hormone …but omits Thymus is-a endocrine gland ►Conciseness ►Redundant inclusion of inferrable axioms ►Consistency – contradictory, duplicated, circular ►endocrine surgery vs endocrine surgeons ►Traumatic unilateral amputation Unilateral traumatic amputation

BioHealth Informatics Group Fitness for Purpose ►Best theories no guarantee of usability or utility ►Lab experiences no predictor of field behaviour ►All for nothing, if user can’t use it ►Interrater variability

BioHealth Informatics Group Fitness for Purpose: Inter-rater variability XXXXXXXXXX XXXXXXXXXXXXXX XXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXX XXXXXX XXXXXXXXXXXX XXXXXXXXXXXX XXXXXXXXXXXX Headcloth Cloth Scarf Model Person Woman Adults Standing Background Brown Blue Chemise Dress Tunics Clothes Suitcase Luggage Attache case Brass Instrument French Horn Horn Tuba

BioHealth Informatics Group Fitness for Purpose: Inter-rater variability ►Miscoding ►Code meaning is inappropriate to thing being described ►Instrument definitely not a french horn ►Missed coding ►Not coding something that could be coded ►No code for the table/platform ►Overcoding ►Code meaning is more detailed than justified ►Can the gender really be determined? ►Undercoding ►Code meaning is less detailed than justified ►Brass Instument vs Tuba

BioHealth Informatics Group Fitness for Purpose What ontological properties.. ►Increase usablility and utility? ►Are a prerequisite for them? ►Decrease usability or utility?

BioHealth Informatics Group Ontology QA How much quality do we need? Perfection is unattainable Trade-offs between quality and… Performance Cost Maintainability Usability Acceptance Utility

BioHealth Informatics Group Need for CQI, not final QA Assure the p rocess not the product Use/Test cases & exemplars REVIEW Explore consequences APPLY QI Algorithm DEVISE QI Algorithm ASSESS Identify problems