1. © University of Manchester Creative Commons Attribution-NonCommercial 3.0 unported 3.0 license Quality Assurance, Ontology Engineering, and Semantic Interoperability wrt Terminologies & Ontologies Alan Rector BioHealth Informatics Group School of Computer Science & Northwest Institute of BioHealth Informatics University of Manchester, Manchester M13 9PL rector@cs.manchester.ac.uk

2. © University of Manchester Creative Commons Attribution-NonCommercial 3.0 unported 3.0 license 2 End-to-end preservation of meaning ►Faithful transmission of information ►The same actions/decisions/treatment where-ever the patient is treated ►Decisions based on the largest pool of research data possible Decisions on Input / output Decisions on output / input

3. © University of Manchester Creative Commons Attribution-NonCommercial 3.0 unported 3.0 license 3 Aspects of Artifacts for Faithful Interoperability wrt terminologies / ontologies ►Competence / coverage - ►does it provide the information? can it answer the relevant questions? Identify value sets? Constrain what is legal? What can be said where? ►Correctness ►with regards to what gold standard? ►Usability - ►can it be used by authors? end users? rearchers? …? ►Reliability ►do users always enter / understand information the same way? the same user? different users? different locales? ►is it always bound to documents / EHRs in the same way? ►Robustness - what errors are introduced in the face of the unexpected / incorrect use? How likely are different types of incorrect usage? ►technical? ►human factors?

4. © University of Manchester Creative Commons Attribution-NonCommercial 3.0 unported 3.0 license 4 Faithfulness & QA at all levels ►Pragmatic ‣ are the same decision taken by all interoperating systems / users? ‣ does everyone use it the same way? (inter-rater reliability)? ‣ do users use it as intended - rater-author consensus? ►Semantic / Ontological ‣ are the intended meanings explicit? correct? attributed to authorities? ‣ do users understand the implications of their entries? queries? Logical / Software -Are the inferences and answers to queries as intended? -Does the logical model match the intended semantics? -What extra-logical representation is needed? How can it be validated? -Is it logically self consistent ? Does extralogical operations conform to spec? ‣ Lexical / linguistic -Are ID and Naming conventions observed? synonyms adequate? -Do labels reflect clinical discourse intuitively? Are they misleading? -Are multilingual/multi-cultural issues addressed? Validated? Anomalies public? -* Syntactic -Does the data model connect properly? - …

5. © University of Manchester Creative Commons Attribution-NonCommercial 3.0 unported 3.0 license 5 QA methods for terminologies / ontologies Human FactorsOntological / Logical / Software Pragmatic End-to-end decision testing inter-rater reliability / repeatability Corpus studies for content coverage Metadata for: provenance, editorial status, versioning & localisation, known issues, mappings, links to external resources, guidelines, … Semantic / Ontological Specification of questions to be asked / answered Authoritative consultation Consistency with external sources Textual definitions & paraphrases Analysis of definitions & hierarchies Logical / Software Challenge testing Usability studies Language generation for user friendly validation of logical structure Logical definitions & descriptions Correctness of inferences, logical hierarchies, & query answering Logical consistency Schema / pattern / criteria conformance testing (unit testing; regression testing) Connectivity testing Lexical / Linguistic Corpus studies for language: synonyms anaphora, variants, … Comprehension studies Back translation studies Language generation Text mining Lexicon structure

6. © University of Manchester Creative Commons Attribution-NonCommercial 3.0 unported 3.0 license 6 Potential actions ►Establish collaboratory for practical testing of interoperability of each component against agreed competencies ►Technical ►Human factors ►End-to-end ►Resource groups to address specific known issues in key vocabularies and standards ►SNOMED, LOINC, ICD, … ►Support Interworking of ICD-11 & SNOMED