“Ontology Measurement and evaluation" mini-series Realism-based Change Management for Quality Assurance in Ontologies and Data Repositories NIST-Ontolog-NCOR, Januari 11, 2007 Werner CEUSTERS, MD Center of Excellence in Bioinformatics and Life Sciences Department of Psychiatry, University at Buffalo, NY, USA http://www.org.buffalo.edu/RTU

Outline A realist view on ontology Coping with changes Tracking changes in ontologies and repositories A calculus for quality assurance of updates

Part 1: A Realist View on Ontology

An unfortunate perception of ‘ontology’ The most widespread view of what an ontology is, is that of ‘an explicit specification of the conceptualization of a domain’ (Gruber), often complemented with the notion of ‘agreement’.

This view on ontology has sad consequences Too much effort goes into the specification business OWL, DL-reasoners, translators and convertors, syntax checkers, ... Too little effort into the faithfulness of the conceptualizations towards what they represent. Pseudo-separation of language and entities “absent nipple” Many ‘ontologies’ and ontology-like systems exhibit mistakes of various sorts.

The remedy: a realist view of the world The world consists of entities that are Either particulars or universals; Either occurrents or continuants; Either dependent or independent; and, relationships between these entities of the form <particular , universal> e.g. is-instance-of, <particular , particular> e.g. is-member-of <universal , universal> e.g. isa (is-subtype-of) Smith B, Kusnierczyk W, Schober D, Ceusters W. Towards a Reference Terminology for Ontology Research and Development in the Biomedical Domain. Proceedings of KR-MED 2006, November 8, 2006, Baltimore MD, USA

Three levels of reality The world exists ‘as it is’ prior to a cognitive agent’s perception thereof; Smith B, Kusnierczyk W, Schober D, Ceusters W. Towards a Reference Terminology for Ontology Research and Development in the Biomedical Domain. Proceedings of KR-MED 2006, November 8, 2006, Baltimore MD, USA

Reality exist before any observation

Reality exist before any observation And also most structures in reality are there in advance.

Three levels of reality The world exists ‘as it is’ prior to a cognitive agent’s perception thereof; Cognitive agents build up ‘in their minds’ cognitive representations of the world; Smith B, Kusnierczyk W, Schober D, Ceusters W. Towards a Reference Terminology for Ontology Research and Development in the Biomedical Domain. Proceedings of KR-MED 2006, November 8, 2006, Baltimore MD, USA

B The ontology author acknowledges the existence of some Portion Of Reality (POR) R

Some portions of reality B Some portions of reality escape his attention. R

Three levels of reality The world exists ‘as it is’ prior to a cognitive agent’s perception thereof; Cognitive agents build up ‘in their minds’ cognitive representations of the world; To make these representations publicly accessible in some enduring fashion, they create representational artifacts that are fixed in some medium. Smith B, Kusnierczyk W, Schober D, Ceusters W. Towards a Reference Terminology for Ontology Research and Development in the Biomedical Domain. Proceedings of KR-MED 2006, November 8, 2006, Baltimore MD, USA

He represents only what he considers relevant B RU1B1 Both RU1B1 and RU1O1 are representational units referring to #1; RU1O1 is NOT a representation of RU1B1; RU1O1 is created through concretization of RU1B1 in some medium. RU1O1 O #1 R

“concept representation” We should not be in the business of Thus ... These concretizations are NOT supposed to be the representations of these cognitive representations; “concept representation” We should not be in the business of

They are representations of the corresponding parts of reality But beware ! These concretizations are NOT supposed to be the representations of these cognitive representations; They are representations of the corresponding parts of reality They are like the images taken by means of a high quality camera;

They are not (or should not be) like the paintings of Salvador Dali Non-canonical (although nice looking) anatomy

Representational artifacts Ideally built out of representational units and relationships that mirror the entities and their relationships in reality. Non-Formalized Formalized Primarily about particulars Progress notes, discharge letters, medical summaries, maps, ... Inventories, referent tracking database, ... Primarily about universals and defined classes Medical textbooks, scientific theories, ... Ontologies, terminologies, ...

Some characteristics of representational units each unit is assumed by the creators of the representation to be veridical, i.e. to conform to some relevant POR as conceived on the best current scientific understanding;

Some characteristics of representational units each unit is assumed by the creators of the representation to be veridical, i.e. to conform to some relevant POR as conceived on the best current scientific understanding; several units may correspond to the same POR by presenting different though still veridical views or perspectives;

Some characteristics of representational units each unit is assumed by the creators of the representation to be veridical, i.e. to conform to some relevant POR as conceived on the best current scientific understanding; several units may correspond to the same POR by presenting different though still veridical views or perspectives; what is to be represented by the units in a representation depends on the purposes which the representation is designed to serve.

Some characteristics of an optimal ontology Each representational unit in such an ontology would designate (1) a single portion of reality (POR), which is (2) relevant to the purposes of the ontology and such that (3) the authors of the ontology intended to use this unit to designate this POR, and (4) there would be no PORs objectively relevant to these purposes that are not referred to in the ontology.

But things may go wrong … assertion errors: ontology developers may be in error as to what is the case in their target domain;

Assertion error B O RU1B1 RU1O1 R #1

But things may go wrong … assertion errors: ontology developers may be in error as to what is the case in their target domain; relevance errors: they may be in error as to what is objectively relevant to a given purpose;

Relevancy error B O RU1B1 RU1O1 R #1

But things may go wrong … assertion errors: ontology developers may be in error as to what is the case in their target domain; relevance errors: they may be in error as to what is objectively relevant to a given purpose; encoding errors: they may not successfully encode their underlying cognitive representations, so that particular representational units fail to point to the intended PORs.

Encoding error B O RU1B1 RU1O1 R #1

Example: medical ‘findings’ and ‘observations’ (1) A particular pathological entity may at a certain time be undetectable by any observation method or technique available to an observer, including the person exhibiting the pathological entity itself.

Example: medical ‘findings’ and ‘observations’ (1) A particular pathological entity may at a certain time be undetectable by any observation method or technique available to an observer, including the person exhibiting the pathological entity itself. A particular observation may produce false results and thus simulate the existence of a pathological entity.

Example: medical ‘findings’ and ‘observations’ (1) A particular pathological entity may at a certain time be undetectable by any observation method or technique available to an observer, including the person exhibiting the pathological entity itself. A particular observation may produce false results and thus simulate the existence of a pathological entity. An observer may observe or fail to observe a detectable particular pathological entity.

On ‘findings’ and ‘observations’ (2) When an observer perceives a particular pathological entity, he might judge it (1) to be an instance of the universal of which it is indeed an instance in reality, (2) to be an instance of another universal (and thus be in error), or (3) he might be not able to make an association with any universal at all. Distinct manifestations of ‘the same type’ may be pathological or not: Singing naked under the shower versus in front of The White House ...

Part 2: Coping with changes

Reality versus beliefs, both in evolution p3 Reality

Reality versus beliefs, both in evolution p3 Reality IUI-#3 O-#0 O-#2 Belief O-#1 = “denotes” = what constitutes the meaning of representational units …. Therefore: O-#0 is meaningless

Reality versus beliefs, both in evolution p3 IUI-#3 O-#0 O-#2 B O-#1 Total ignorance: e.g. a disease (U1) already exist but we have no clue about it

Reality versus beliefs, both in evolution p3 IUI-#3 O-#0 O-#2 B O-#1 False belief in the existence of a type, e.g. ‘unicorn’, ‘diabolic possession’ Note: happens also at the level of particulars: e.g. the planet ‘Vulcan’

Reality versus beliefs, both in evolution p3 IUI-#3 O-#0 O-#2 B O-#1 The coming into existence of a new universal remains unnoticed: e.g. ‘AIDS’ existed before being discovered.

Reality versus beliefs, both in evolution p3 IUI-#3 O-#0 O-#2 B O-#1 The coming into being of a new particular remains unnoticed: e.g. John Doe’s colonic polyp, which from that time on, is an instance of U1.

Reality versus beliefs, both in evolution p3 IUI-#3 O-#0 O-#2 B O-#1 An advance in science: the existence of U1 is acknowledged.

Reality versus beliefs, both in evolution p3 IUI-#3 O-#0 O-#2 B O-#1 The existence of John Doe’s benign colonic polyp is discovered, however, without being recognized as such. Rather, it is believed to be an instance of what in reality is a fantasy.

Reality versus beliefs, both in evolution p3 IUI-#3 O-#0 O-#2 B O-#1 Another advance in science: the ‘concept’ O-#0 is rightfully abandoned, necessitating therefor to reconsider of what p3 must be believed to be an instance of.

Reality versus beliefs, both in evolution p3 IUI-#3 O-#0 O-#2 B O-#1 It is, rightfully believed that p3 is an instance of U1. It raises, amongst other things, the question to what point in history this belief can be extended.

Reality versus beliefs, both in evolution p3 IUI-#3 O-#0 O-#2 B O-#1 p3 changes from a benign into a malignant tumor, at a time that science did not discover malignancy yet. p3 is now wrongly believed to be an instance of U1.

Reality versus beliefs, both in evolution p3 IUI-#3 O-#0 O-#2 B O-#1 Advance in science: “malignancy” is discovered. However, that it applies to John Doe’s polyp has not yet been noticed.

Reality versus beliefs, both in evolution p3 IUI-#3 O-#0 O-#2 B O-#1 John Doe’s polyp becomes recognized as an instance of a malignant tumor.

Reality versus beliefs, both in evolution p3 IUI-#3 O-#0 O-#2 B O-#1 John Doe’s polyp was irradiated and believed to have vanished, while in reality, it isn’t.

Reality versus beliefs, both in evolution p3 IUI-#3 O-#0 O-#2 B O-#1 John Doe is lucky: his tumor indeed disappeared. His physicians who believed it was already gone, are lucky also: they escape a law suite.

Reality versus beliefs, both in evolution p3 IUI-#3 O-#0 O-#2 B O-#1 For ‘utilitarian’ reasons, the “pragmatic engineers” remove malignant tumors from their ontology: if it is not believed to exist, you can’t get law suites for failures in recognizing instances.

Some other possible situations IUI-#3 IUI-#3 O-#0 O-#2 B O-#1 A particular is believed to exist longer than it really does. e.g. “Elvis is not dead”, or the innumerous EHRs that state the patient taking some drug while he stopped.

And in this, I thus far ignored … p3 IUI-#3 O-#0 O-#2 B O-#1 Relationships amongst universals (R) or beliefs therein (B)

Why this story ? It shows … the complex interrelationships between What is the case; What we know about what is the case; What parts about what we know that is the case we wish to refer to in ontologies and repositories. the need to update ontologies and repositories in line with various sorts of changes.

Mistakes, discoveries, being lucky, having bad luck p3 IUI-#3 O-#0 O-#2 B O-#1

Mistakes, discoveries, being lucky, having bad luck p3 IUI-#3 O-#0 O-#2 B O-#1

Mistakes, discoveries, being lucky, having bad luck p3 IUI-#3 O-#0 O-#2 B O-#1

Mistakes, discoveries, being lucky, having bad luck p3 IUI-#3 O-#0 O-#2 B O-#1

A crucial question … Do we have some means to assess how good we are doing in our understanding of reality ? Some might argue: no, because every representational unit will always rest on a belief ! My belief is: yes, if one keeps track of the reasons, in function of the three levels discussed earlier, for which one’s beliefs have changed.

Part 3: Tracking Changes in Representations Ceusters W, Smith B. Towards A Realism-Based Metric for Quality Assurance in Ontology Matching. Forthcoming in Proceedings of FOIS-2006, Baltimore, Maryland, November 9-11, 2006. Ceusters W, Smith B. A Realism-Based Approach to the Evolution of Biomedical Ontologies. Forthcoming in Proceedings of AMIA 2006, Washington DC, November 11-15, 2006.

Remember our 3 fundamentally different levels the reality on the side of the patient; the cognitive representations of this reality embodied in observations and interpretations on the part of clinicians and others; the publicly accessible concretizations of such cognitive representations in representational artifacts of various sorts, of which ontologies and terminologies are examples.

Example: a person (in this room) ’s phenotypic gender Reality: Male Female Cognitive representation [male] [female] In the EHR: “male” “female” “unknown” Other types of phenotypic gender ?

The specification bias in ontology evolution (1) Ontology versions exhibit differences of the following sorts: Add a subtree Delete a subtree Move a subtree to a different location Move a set of sibling classes to a different location Create a new abstraction and move a set of siblings down in a class hierarchy, creating a new superclass. Delete a class, moving its subclasses to become subclasses of its superclass. Split a class Merge classes Noy, N.F., Kunnatur, S., Klein, M., and Musen, M.A. Tracking changes during ontology evolution. In Proceeding of the 3rd International Semantic Web Conference (ISWC2004), Hiroshima, Japan, November 2004.

The specification bias in ontology evolution (2) Yaozhong David Liang. Enabling Active Ontology Change Management within Semantic Web-based Applications PhD thesis October 2, 2006

They don’t care about the reasons for the changes changes in the underlying reality (does the appearance or disappearance of an entry in a new version of an ontology relate to the appearance or disappearance of entities or of relationships among entities?); changes in our scientific understanding; reassessments of what is relevant for inclusion in an ontology; encoding mistakes introduced during ontology curation (for example through erroneous introduction of duplicate entries reflecting lack of attention to differences in spelling).

Key requirement for versioning Any change in an ontology or data repository should be associated with the reason for that change to be able to assess later what kind of mistake has been made !

Example: a person’s gender in the EHR In John Smith’s EHR: At t1: “male” at t2: “female”

Example: a person’s gender in the EHR In John Smith’s EHR: At t1: “male” at t2: “female” What are the possibilities ?

Example: a person’s gender in the EHR In John Smith’s EHR: At t1: “male” at t2: “female” What are the possibilities ? Change in reality:

Example: a person’s gender in the EHR In John Smith’s EHR: At t1: “male” at t2: “female” What are the possibilities ? Change in reality: transgender surgery

Example: a person’s gender in the EHR In John Smith’s EHR: At t1: “male” at t2: “female” What are the possibilities ? Change in reality: transgender surgery change in legal self-identification

Example: a person’s gender in the EHR In John Smith’s EHR: At t1: “male” at t2: “female” What are the possibilities ? Change in reality: transgender surgery change in legal self-identification Change in understanding: it was female from the very beginning but interpreted wrongly

Example: a person’s gender in the EHR In John Smith’s EHR: At t1: “male” at t2: “female” What are the possibilities ? Change in reality: transgender surgery change in legal self-identification Change in understanding: it was female from the very beginning but interpreted wrongly Correction of data entry mistake: it was understood as male, but wrongly transcribed

Ways representational units do or do not refer OE: objective existence; ORV: objective relevance; BE: belief in existence; BRV: belief in relevance; Int.: intended encoding; Ref.: manner in which the expression refers; G: typology which results when the factor of external reality is ignored. E: number of errors when measured against the benchmark of reality. P/A: presence/absence of term. 72

Ways representational units do or do not refer OE/BE value pairs Y/Y: correct assertion of the existence of a POR; Y/N: lack of awareness of a POR, reflecting an assertion error; N/N: correct assertion that some putative POR does not exist ; N/Y: the false belief that some putative POR exists. 73

Ways representational units do or do not refer Ref.: manner in which the expression refers R+: the encoding of the belief is correct R: the encoding is incorrect because it does not refer R-: it does refer, but to a POR other than the one which was intended. 74

Typology of expressions included in and excluded from a representation in light of relevance and relation to external reality 75

Typology of expressions included in and excluded from a representation in light of relevance and relation to external reality 76

Typology of expressions included in and excluded from a representation in light of relevance and relation to external reality 77

Typology of expressions included in and excluded from a representation in light of relevance and relation to external reality 78

Typology of expressions included in and excluded from a representation in light of relevance and relation to external reality 79

Valid presence Valid absence 80

Unjustified presence Unjustified absence 81

But sometimes you get lucky … Unjustified presence Unjustified absence But sometimes you get lucky … 82

Possible evolutions through versions

Possible evolutions through versions 84

Example 1: An entity ceases to exist 85

Example 2: An entity becomes irrelevant 86

Example 3: A relevant entity comes into existence 87

Updating is an active process authors assume in good faith that all included representational units are of the P+1 type, and all they are aware of, but not included, of A+1 or A+2. If they become aware of a mistake, they make a change under the assumption that their changes are also towards the P+1, A+1, or A+2 cases. Thus at that time, they know of what type the previous entry must of have been under the belief what the current one is, and the reason for the change.

Part 4: A Calculus for Update Quality

This leads to a calculus … NOT: to demonstrate how good an individual version of an ontology is, But rather to measure how much it improved (hopefully) as compared to its predecessors. Principle: recursive belief revision

Backward belief revision over time Reality: a POR exists and is not relevant R P Beliefs At t about t -2 At time t, the authors of an ontology correctly perceive the existence of some universal, but consider it relevant while it isn’t, and they make an encoding error such that the representational unit does not refer. There is thus a -2 error with respect to reality, but this remains, of course, unknown.

Backward belief revision over time Reality: a POR exists and is not relevant R P Beliefs At t about t -2 At t+1 about t+1 At t+1 about t At t+1, they correct the encoding mistake, which forces them to believe that at t, the unit-reality configuration was of type P-4 rather than P+1.

Backward belief revision over time Reality: a POR exists and is not relevant R P Beliefs At t about t -2 At t+1 about t+1 At t+1 about t -1 -1 Although they believe that the current situation is P+1, it is in reality P-6, where it was P-7 before. The real error is now -1, while the perceived error with respect to t is also -1

Backward belief revision over time Reality: a POR exists and is not relevant R P Beliefs At t about t -2 At t+1 about t+1 At t+1 about t -1 -1 At t+2, the authors believe that the posited POR in fact does not exist

Backward belief revision over time Reality: a POR exists and is not relevant R P Beliefs At t about t -2 At t+1 about t+1 At t+1 about t -1 -1 At t+2 about t+2 At t+2 about t+1 At t+2 about t -1 -3 -5 95

Can this be implemented ? Manual burden is low: documenting the reason for a change  clicking one radio button. The change of belief revisions is automatically computable from the table shown earlier.

Using this approach for sampling large ontologies Future directions Using this approach for sampling large ontologies Integration of confidence levels Replacing the Y/N dichotomy for beliefs A perhaps more elaborate way of counting errors. Use as a tool for educational purposes to compare the beliefs of various stakeholders, not the least concerning relevance (clinicians, biologists, informaticians, …)