An Evaluation of a Methodology for Specification of Clinical Guidelines at Multiple Representation Levels Student :Erez Shalom Supervisors: Prof. Yuval Shahar Dr. Meirav Taieb-Maymon

ISE Dep. Seminar 26/4/06 Background Methods Results Conclusions Future Directions Talk Roadmap :

ISE Dep. Seminar 26/4/06 Clinical Guidelines Textual documents describing “state of the art” patient management A powerful method to standardize and improve the quality of medical care Usually specify diagnostic and/or therapeutic procedures

ISE Dep. Seminar 26/4/06 Subsections Describing patient diagnosis and treatment

ISE Dep. Seminar 26/4/06 The Need for Automation of Clinical Guidelines Automatic support provides:  Visual specification  Search and retrieval  Application of a GL  Retrospective quality assurance However: Most GLs are text based and electronic inaccessible at the point of care

ISE Dep. Seminar 26/4/06 The Required Infrastructure A machine-comprehensible GL representation ontology (e.g., Asbru ontology) Runtime GL application and QA tools  A preliminary engine, namely, Spock was already developed in our lab by [Young,2005] Support for a gradual structuring of the GL (from text to an executable code)

ISE Dep. Seminar 26/4/06 The Structuring Process The Guideline as a tree of plansThe Guideline as a text document Involves 2 main types of knowledge: Procedural knowledge – e.g. Regimen A for administer the two medications in parallel In parallel D D Declarative knowledge - e.g. 2 g IV Regimen A Cefotetan Doxycline P PP

ISE Dep. Seminar 26/4/06 Sample GL Modeling Methods Knowledge Acquisition toolMethod A Protégé-based interface EON and SAGE, Prodigy, GLIF ArrezoPROforma GEM-CutterGEM "CG_AM" graphical interfaceGLARE NEWGUIDEGUIDE Asbru-View, GMT, StepperAsbru

ISE Dep. Seminar 26/4/06 Expert Physician Knowledge Engineer The Hybrid Representation Model Gradually structuring the GL using increasingly formal representation levels  Implemented as part of the Digital Electronic Guideline Library) DeGeL)  Used within the URUZ GL markup tool

ISE Dep. Seminar 26/4/06 Asbru- the Underlying Guideline- Representation Ontology Conditions KR-Class (e.g., the filter condition, and the abort condition) Plan-body KR-Class for the GL’s Control structures (e.g., sequential, concurrent, and repeating combinations of actions or sub- guidelines), GL’s Goals KR-Class (e.g. process and outcome intentions), Context KR-Class of the activities in the GL (e.g. actors, clinical-context). Includes semantic Knowledge Roles (KRs) organized in KR-Classes such as:

ISE Dep. Seminar 26/4/06 URUZ (I): Specification of declarative knowledge Expert physician Selects “filter condition” knowledge role

ISE Dep. Seminar 26/4/06 URUZ (cont’d) : Specification of Procedural Knowledge Expert physician decomposing the GL into tree of plans

ISE Dep. Seminar 26/4/06 GL Specification: Core Issues √ Expert Physicians (EPs) - Knowledge Engineers (KEs) collaboration √ Incremental Specification √ Treatment of Multiple Ontologies √ Distributed Collaboration and Sharing √ Text Based Source √ Knowledge Conversion

ISE Dep. Seminar 26/4/06 Several unresolved issues:  Definition of the necessary steps for the GL specification process  Use optimally of EPs and KEs in the process  Evaluation is crucial for quantify the markups quality To Achieve high quality of markups there is a Need for: An overall process of guideline specification A complete evaluation methodology

ISE Dep. Seminar 26/4/06 √ Background Methods Results Conclusions Future Directions Talk Roadmap

ISE Dep. Seminar 26/4/06 The Overall Process of Guideline Specification The activities in the markup process include three main phases : 1) Preparations before the markup activities 2) Actions during the Markup activities 3) After Markup activity

ISE Dep. Seminar 26/4/06 A Methodology Specification of Clinical Guidelines Creating a consensus is a crucial, mandatory step before markup

ISE Dep. Seminar 26/4/06 The Importance of Using an Ontology-Specific Consensus (OSC) An OSC is a structural document that describes schematically the clinical directives of the GL Described by the semantic of the specification ontology Prevent disagreement and a great deal of variability among the EPs

ISE Dep. Seminar 26/4/06 Methodology for Creation of OSC The OSC is created in a iterative fashion by performing the following steps 1. First, we create a preliminary structure of the clinical pathway 2. The KE adds procedural, control structure 3. The KE adds declarative concepts for each defined step 4. After some iteration of steps 2 and 3, an OSC is formed

ISE Dep. Seminar 26/4/06 The second stage in forming a consensus

ISE Dep. Seminar 26/4/06 Evaluation Design Amount of Expertise The acquired knowledge domain The Ontology Specific Consensuses The Gold Standard markup for each GL The Markups for each GL The Evaluation of markups Considered some specific Criteria :

ISE Dep. Seminar 26/4/06 Evaluation Design (cont’d) Three GLs in different domains were used :  Pelvic Inflammatory Disease (PID)  Chronic Obstructive Pulmonary Disease (COPD)  Hypothyroidism(HypoThyrd)

ISE Dep. Seminar 26/4/06 Evaluation Design (cont’d)

ISE Dep. Seminar 26/4/06 Research Questions Is markup feasible by EPs? Is there a difference between the EPs editing the same GLs, and same EPs editing difference GLs? Is there a difference between the KRs across all EPs? Is there a difference in the amount of errors when using different OSC?

ISE Dep. Seminar 26/4/06 Evaluation of markups  Subjective Measures - Questionnaires were administered for finding the EPs attitude regarding the specification process  Objective Measures – in two scales (compared to the GS): * Completeness of the markup * Correctness of the markup

ISE Dep. Seminar 26/4/06 The Objective Measures - Completeness

ISE Dep. Seminar 26/4/06 * Clinical Measure (CM) – measure the clinical correctness of the content * Asbru Semantic Measure (ASM) - measure the semantics correctness of the content ( Asbru semantic in our case) The Objective Measures - Correctness

ISE Dep. Seminar 26/4/06 Resolution of Measure Mean (weighted) Quality Score (MQS) for: GLs - to find common trends in a GL, and in all GLs EPs - to find trends in between the markups of the EPs across the same GL and between GLs KRs - to find trends in a specific KR type and common trends across KRs and KR classes across one markup, GL and in all GLs

ISE Dep. Seminar 26/4/06 General errors classified into two types, and thus into two corresponding scales:  Clinical errors:  Clinical content not accurate  Clinical semantics not well specified  Clinical content not complete.  Asbru semantics errors:  Asbru semantics content not accurate  Asbru semantics content not well specified  The content does not includes mappings to standard terms  The necessary knowledge is not defined in the guideline knowledge when it should be. The Objective Measures – Types of Errors

ISE Dep. Seminar 26/4/06 Specific errors for each KR Type a specific error, for example :  Conditions /Intentions KRs:  There are no And/Or operators between the different criteria.  Simple Action Plan-Body Type:  Has no text content describing the plan  Has no single atomic action semantics with clear specification and description for the action to be performed.  Plan Activation Plan-Body Type:  Plan name is not defined  Defined plan does not exist in DeGeL. The Objective Measures – Types of Errors

ISE Dep. Seminar 26/4/06 The Markup-Evaluation Tool

ISE Dep. Seminar 26/4/06 The Markup-Evaluation Tool (Cont’d)

ISE Dep. Seminar 26/4/06 The Markup-Evaluation Tool (Cont’d)

ISE Dep. Seminar 26/4/06 √ Background √ Methods Results Conclusions Future Directions Talk Roadmap

ISE Dep. Seminar 26/4/06 Results – Subjective Measures ResultStagePurpose Using their medial knowledge and their understanding of the specification ontology (Asbru, in our case) Vs. specification Tools after creating the OSC Finding the Aspects that most helped the EPs' when creating an OSC 1 Specification Tools is considered as more helpful after markupFinding the Aspects that most helped the EPs' making a markup 2 Declarative KRs are more easy to understand (such as filter condition) before markupFinding how well The EPs Understand Asbru KRs 3 Procedural KRs are more easy to structure (such periodic plan) after markupFinding what were the difficulties of the EPs' in structuring the Asbru KRs 4 SUS=47 ; Not Usable!after markupSystem Usability Scale (SUS) for URUZ 5 Non significant correlation between results 1 and 2 Significant correlation between results 3 and 4

ISE Dep. Seminar 26/4/06 Results – Objective measures Mean Completeness for all markups of EPs of 91% All markups of EPs has significant (P<0.05) proportion of scores of 1 higher than 0.33 (some even higher then 0.75) Markup is feasible by EPs Number of specified plans: Measures Summary:

ISE Dep. Seminar 26/4/06 Results –Difference between EPs Any EP can perform markup with high completeness There is wide variability between the EPs in the correctness measure with a range of [0.13,0.58] on a scale of [-1,1] √ Difference in correctness measure between different GLs editing the same EPs 4 Difference in Correlation measure between EPs editing the same GLs in most GLs (except the PID) 3 √ Difference in correctness measure between EPs editing the same GLs in most GLs (except the Hypo) 2 nonsignificant (P>0.05) Difference between the proportions of completeness measure between EPs editing the same GLs 1 significant (P<0.05) Issue √ √

ISE Dep. Seminar 26/4/06 Results – Difference between KRs There was significant difference (P<0.05) between homogenous groups of KRs EPs has difficulty to structure procedural KRs than declarative ones

ISE Dep. Seminar 26/4/06 Results – Types of errors The differences in total between the three GLs were highly significant in a proportion test (P<0.001) The more detailed and structured the OSC was, the lower the total number of errors committed by the EPs for each KR

ISE Dep. Seminar 26/4/06 √ Background √ Methods √ Results Conclusions Future Directions Talk Roadmap

ISE Dep. Seminar 26/4/06 Four main aspects: Creation of an Ontology-Specific Consensus (OSC) The essential aspects needed to learn to support the specification process by EPs The medical and computational qualifications needed for specification The characteristics of the KA tool needed for this kind of specification

ISE Dep. Seminar 26/4/06 Creation of an Ontology-Specific Consensus (OSC) Should be made as detailed as possible, including all relevant procedural and declarative concepts The OSC is independent of the specification tool Saving the OSCs in an appropriate digital library for re-using and sharing

ISE Dep. Seminar 26/4/06 The Essential Aspects Needed to Learn to Support the Specification Process by EPs creating an OSC and performing the markups are two different tasks which require teaching two different aspects Teaching the “difficult” KRs in particular, the procedural KRs Short test should be administered before the EPs perform markups A help manual and a small simulation of marking up a GL should be included in the teaching session

ISE Dep. Seminar 26/4/06 The Medical and Computational Qualifications Needed for Specification Senior EPs and KEs together should work on the tasks of selecting a GL for specification and making the OSC Any EP (senior, non-senior or a general physician) can structure the GL's knowledge in a semiformal representation completely To specify it correctly, a more available EP should be selected, perhaps from among residents, interns or even students

ISE Dep. Seminar 26/4/06 The Characteristics of the KA Tool Needed for This Kind of Specification A robust, graphical, highly usable framework is needed More intuitive, graphic, user friendly interfaces should be used for acquiring the “difficult” KRs, especially the procedural ones Need to bridge the gap between the initial structuring of the EP and the full semantics of the specification language GESHER - A Graphical Framework for Specification of Clinical Guidelines at Multiple Representation Levels

ISE Dep. Seminar 26/4/06 Limitations and Advantages of the research Small of the number of EPs and GLs, But, in fact, 196 sub-plans and 326 KRs in total were structured by all of the EPs together in all markups Lack of careful measurement of the required time, but, obtain more realistic results, since the interaction with most of the EPs took place in their own "playground"

ISE Dep. Seminar 26/4/06 √ Background √ Methods √ Results √ Conclusions Future Directions Talk Roadmap

ISE Dep. Seminar 26/4/06 The GESHER’s Main Features  User friendly graphical client application  Support specification at multiple representation levels  Support to multiple specification languages (GL ontologies)  Access centralized resources such as DeGeL and a knowledge base

ISE Dep. Seminar 26/4/06 GESHER: Semi-Structured Level

ISE Dep. Seminar 26/4/06 GESHER(II) :Semi-Formal Widgets

ISE Dep. Seminar 26/4/06 √ Background √ Methods √ Results √ Conclusions √ Future Directions Talk Roadmap

ISE Dep. Seminar 26/4/06 Summary The need for gradual GL specification  Making an ontology-specific consensus as first step  Use a well defined methodology for the overall process Markup is feasible by EPs Any EP can perform markup with high completeness We should use methodology for increase quality of markups Use GESHER as the new framework for specification Ongoing new research is being conducted (Pre-Eclampsia GL) based on this research results

ISE Dep. Seminar 26/4/06 Research Publications סביבת עבודה גראפית במס' רמות ייצוג, הכנס הישראלי למ"מ רפואיות 2005 Shalom, E. and Shahar Y. (2005). A Graphical Framework for Specification of Clinical Guidelines at Multiple Representation Levels. AMIA Annual Fall Symposium, Washington DC, USA האם רופאים מסוגלים להבנות ידע רפואי? הכנס הישראלי למ"מ רפואיות 2006 Shalom E, Shahar Y, Young O, Bar G, Taieb-Maimon M, Yarkoni A, B.Martins S, Vaszar L, K.Goldstein M, Liel Y, Leibowitz A, Marom T, and Lunenfeld E. (2006) A Methodology for Evaluation of A Markup-Based Specification of Clinical Guidelines Submitted to AMIA, Washington DC, USA Shalom E, Shahar Y, Young O, Bar G, Taieb-Maimon M, Yarkoni A, B.Martins S, Vaszar L, K.Goldstein M, Liel Y, Leibowitz A, Marom T, and Lunenfeld E.(2006) The Importance of Creating an Ontology-Specific Consensus Before a Markup-Based Specification of Clinical Guidelines, Submitted to ECAI06,Tronto, Italy JAMIA journal paper is in preparation

ISE Dep. Seminar 26/4/06 Acknowledgments Prof. Yuval Shahar and Dr. Meirav Taib-Maymon All Medical informatics research center members Our colleagues at Soroka’s university medical center : Prof. Eitan Lunenfeld, Dr. Avi Yarkoni, Dr. Guy Bar, Prof. Yair Liel and Dr. Tal Marom Our colleagues at Stanford and VA hospital: Drs. Mary Goldstein, Susana Martins, Lawrence Basso, Herbert Kaizer, Laszlo Tudor Vaszar NLM award No LM Contact info : Visit our web site :

ISE Dep. Seminar 26/4/06 Questions?

ISE Dep. Seminar 26/4/06 EPs create a Clinical consensus : EPs and KE adds procedural knowledge : EP+KE add declarative knowledge : Ontology- specific consensus Doxycycline Metronidazole Order :parallel Filter condition for drug : Is patient not sensitive to Doxycycline and the drug available? Give Doxycycline 100 mg orally or IV every 12 hours Plus Metronidazole 500 mg IV every 8 hours Methodology for creating an Ontology-Specific Consensus

ISE Dep. Seminar 26/4/06 Textual Source

ISE Dep. Seminar 26/4/06 The first stage in forming a consensus

ISE Dep. Seminar 26/4/06 The second stage in forming a consensus

ISE Dep. Seminar 26/4/06 The third stage in forming a consensus

ISE Dep. Seminar 26/4/06 Results – Subjective measures

ISE Dep. Seminar 26/4/06

Results – Objective measures Mean Completeness for all markups of EPs of 91% All markups of EPs has significant (P<0.05) proportion of scores of 1 higher than 0.33 (some even higher then 0.75 Markup is feasible by EPs

ISE Dep. Seminar 26/4/06