1. Semantics models and ontologies in surgical data science Bernard Gibaud Equipe MediCIS, LTSI INSERM 1099, Rennes, France 1

2. Overview Context and motivations Semantic models and ontologies Toward a core ontology of Surgical Process Models (SPM) Conclusion 2

3. Context and Motivations 3

4. Use of ontologies in Surgical data science Many applications – Development of context-aware systems (phase /activity recognition, etc.) – Management of human - surgical robot interaction – Development of simulation systems for training – Creation of annotated datasets for algorithmic development and performance assessment – Semantic annotation of clinical datasets in surgery – … 4

5. Use of ontologies in Surgical data science All these applications need vocabulary and information models to describe surgical processes – Formal: because many of the applications involve some form of reasoning – Standard: because the applications may involve multiple components that need to be able to interoperate, or will involve some learning components, that depend on annotated datasets 5

6. Focus on intraoperative processes 6 Surgery Effectors Sensors Control Surgery recorder Signals and images Annotations Procedure log

7. Challenge: consistency of vocabulary and information models 7 Procedure plan / Procedure model Control software e. g. Phase recognition, command of effectors Actions of effectors e.g. robots Procedure log Annotations of signals and images Human control (Human  Machine) Information (Machine  Human)

8. Semantic models and ontologies 8

9. Ontologies: origin In Philosophy : ontology is a branch of metaphysics – that studies “being“ and their basic categories In Computer science, ontologies have two main origins – research in AI (Knowledge Representation) – web technology (need to semantically annotate the resources available on the web)  has led to the « Semantic Web » 9

10. Ontologies in computer science Ontologies are engineering artefacts consisting of: – a vocabulary modeling a set of real-world phenomena in a given domain – an explicit and formal (i.e.machine-processable) specification of the intended meaning of this vocabulary involves an “is a“ relation between terms of this vocabulary (building a taxonomy) involves constraints capturing (a part of) the semantics of this intended meaning Any ontology should gather some consensus in a community 10 Origin: I Horrocks, 2007

11. Ontologies: origin In medical informatics – Controlled vocabulary: use limited of selected terms – Terminology: controlled vocabulary with additional information and relations between terms (‘broader than’, ‘narrower than’) ; example: Wordnet – Taxonomy: controlled vocabulary organized along a ‘is a’ relationship (subsumption) – Ontology: taxonomy with additional relations, and formal definitions 11

12. 12 Ontology languages Most ontologies are represented in OWL (the Web Ontology Language) OWL is a W3C standard – whose syntax relies on RDF/RDFS – based on Description Logics

13. Description Logics: basics The domain of interest is modeled as a set of – Concepts (i.e. classes of domain entities) – Roles (that denote properties or relations) – Individuals and assertions Paris “is a“ city John “has parent“ Mary Axioms may be associated to the definition of classes to capture (part of) their semantics 13 Called T-Box Called A-Box

14. Added value of ontologies 1. Clarifies the vocabulary thanks to associated axioms 2.Enables automating the control of consistency thanks to the use of reasoners 3.Provides enhanced features in terms of querying thanks to the knowledge embedded in the ontology standard query language : SPARQL 1.1 14

15. Use of ontologies: a three-step process 1. Definition of terminology (i.e. the T-Box) – Classes, Object properties and Data properties 2. Use of this terminology in the representation of data – Representation of the data as instances of the ontology’s classes – Use of the object properties of the ontology to link the instances –  usually encoded as an RDF dataset (RDF graph) 3. Such RDF graph may then be queried (using the SPARQL language) 15

16. Use of ontologies in Surgical Data Science 16 Annotation dataset Annotation process Data to be semantically annotated, e.g.surgical video e.g., description of surgical actions, actors, affected objects, instruments, etc. Ontology Definition of the ontology human community Query process User or application query Selected data of interest

17. Use of ontologies in Surgical Data Science 17 RDF annotation dataset IRI _xx01 _xx02 _xx03 IRI Prop IRIs isA _xx01 _xx02 _xx03 relx rely relz _yy01 _yy02 _yy03 Annotation process Data to be semantically annotated, e.g. videos, instruments’ movements, physiological data, etc. Label ‘xx’ ‘yy’ ‘zz’ IRI Ontology Definition of the ontology human community Query process User or application query (SPARQL query) RDF sub-graph

18. Toward a core ontology of Surgical Process Models 18

19. First ontologies of SPM Rossi Mori et al. An ontological analysis of surgical deeds, AIME 97. CEN EN 1828:2002, Health informatics — Categorial structure for classifications and coding systems of surgical procedures, 2002. Jannin et al. Model of surgical procedures for multimodal image-guided neurosurgery, CAS 2003. Neumuth et al. Modeling surgical processes: a four- level translational approach, AIM 2011. 19

20. Origin: S3PM project Synthesis and Simulation of Surgical Process Models Simulation scenari for the training of scrub nurses using Virtual Reality Use of videos of real procedures Need for a language to describe and annotate the videos 20

21. Ontology development S3PM-related goal – to create a common ontology to meet the S3PM project’s requirements OntoSPM-related goal – To create a core ontology of surgical processes, suitable for S3PM as well as other projects 21

22. Application 1 InstrumentBody partActionRole Application n InstrumentBody partActionRole … ONTOSPM (generic) HumanInstrumentBody partActionRole 22 Modularity

23. OntoSPM: Motivations Why an open core ontology of SPM ? – To facilitate the work of those who need ontologies of SPM – To encourage the creation of a standard for surgical procedure description 23

24. OntoSPM: Main design choice Reuse of relevant existing ontologies – e.g. FMA (anatomy), MPATH (pathology), IAO/OBI (information content entities), PATO (qualities), UO (units of measure) – Designed according to a realist viewpoint Use of a foundational ontology: BFO2 (+RO) Use of OWL DL as an ontology language 24

25. Domain currently covered Surgical processes modeled as bfo:process – real-life processes, i.e. processes that have occurred in reality Emphasis was primarily put on actions, and on the entities participating in the actions, each mode of participation being modeled using roles (bfo:role) 25

26. Current status A preliminary version of OntoSPM exists – not broadly distributed, yet – currently in use in limited applications S3PM project (MediCIS, Rennes) Surge Track software (b<>com, Rennes) LapOntoSPM (KIT, Karlsruhe) From this starting point – Setup a collaboration to create an extended version – a workshop was organized to explore how such collaboration should be setup 26

27. First OntoSPM workshop, 28-29 April, 2016 in Rennes Elena de Momi (Politecnico Milano, Italy): Ontologies in surgical assistance systems Darko Katic (KIT Karlsruhe, Germany): Surgical Phase Recognition in Laparoscopy Guang Zhong Yang (Imperial College London, UK): Ontologies for enhanced man-robot collaboration Alexandre Moreau Gaudry, Sophie Silvent (UJF Grenoble, France): Ontology for CAS technology assessment Juliane Neumann (ICCAS, Leipzig, Germany): Ontologies for surgery at ICCAS – Experiences from 2005 to 2015 Keno März, Lena Maier-Hein (DKFZ, Heidelberg, Germany): Holistic data modeling and decision making in a large collaborative research center Paulo J.S. Gonçalves (Polytechnic Institute of Castelo Branco, Castelo Branco, Portugal): Knowledge representation in robotic orthopedic surgery and IEEE-RAS Ontologies for Robotics and Automation Bernard Gibaud (Inserm, Rennes): OntoSPM: a core ontology for surgical process models 27 Invited presentations

28. First OntoSPM workshop, 28-29 April, 2016 in Rennes Discussion on future collaboration Content of collaboration – Domain of interest of collaboration – Scope management policy – What is developed and shared: ontology, software – Reuse of non-open non-free ontologies SNOMED CT Mode of organization – standards (e.g. DICOM) or more flexible mode Funding Distribution policy 28

29. First OntoSPM workshop, 28-29 April, 2016 in Rennes Major conclusions : content (1/2) Sufficiently broad scope in terms of applications Linked to clinical needs, especially regarding safety Not only ontologies but also software – Software supporting the ontology design – Annotation software – Toolkits for developing application software 29

30. First OntoSPM workshop, 28-29 April, 2016 in Rennes Major conclusions : content (2/2) Efforts towards a core ontology Based on a foundational ontology Try to use SNOMED CT 30

31. First OntoSPM workshop, 28-29 April, 2016 in Rennes Major conclusions: mode of organization a flexible model of co-development is needed at the beginning then, standardization – e.g. through DICOM WG24 31

32. First OntoSPM workshop, 28-29 April, 2016 in Rennes Major conclusions : funding Funding is necessary – to pay staff, for coordination and maintenance Possible options are – EU COST Action – EU Coordination action 32

33. OntoSPM Collaborative Action Launched beginning of June 2016 – Setup of of Wiki platform the current version of OntoSPM is online – The main research groups who contributed to the workshop were invited to join 6 groups responded positively – If you want to contribute and join the action: bernard.gibaud@univ-rennes1.fr 33

34. Conclusion Representing the semantics of shared information is a major ingredient for successful collaborative surgical data science Semantic web technologies certainly provide valuable tools in this context Defining suitable ontologies is challenging, from both a scientific and an organisational viewpoint. 34

35. Acknowledgements CominLabs Labex for support OntoSPM contibutors – Pierre Louis Hénaux – Pierre Jannin – Darko Katic – Cédric Pénet – Javier Rojas Balderrama All participants of the First OntoSPM Workshop 35

36. Thank you for your attention QUESTIONS ? 36