1. Knowledge-Based Interpretation, Visualization, and Exploration of Time-Oriented Medical Data Yuval Shahar, M.D., Ph.D. Medical Informatics Center Information Systems Engineering Ben Gurion University Beer Sheva, Israel And Departments of Medicine and Computer Science Stanford Medical Informatics Stanford University Stanford, CA, USA

2. Time in Natural Language From— “Mr. Jones was alive after Dr. Smith operated on him” Does it follow that— “Dr. Smith operated on Mr. Jones before Mr. Jones was alive?” Is Before the inverse of After?

3. Timing is Everything : Applications of Temporal Reasoning Natural-language processing (e.g., medical record understanding) Planning (e.g., robot planning, therapy planning) Causal reasoning (e.g., diagnosis) Archeology (e.g., seriation) Psychology (e.g., developmental beahvioral psychology) Scheduling (e.g., optimal ordering) Circuit design (e.g., sequential circuits) Software design (e.g., parallel processing, communication, verification) Other, not necessarily time-oriented, domains where interval algebra is useful, such as molecular biology (e.g., arrangement of DNA segments along a linear DNA chain) and evaluation of spatiotemporal traffic-control patterns

4. Allen's Temporal Logic (1981–1984) Only temporal intervals - no instantaneous events 13 basic (binary) interval relations (b,a,eq,o,oi,s,si,f,fi,d,di,m,mi) and transitivity relations between them Properties hold over every subinterval of an interval —> Holds(p, T) e.g., "Patient1's skin was blue throughout sunday" Events hold only over an interval and not over any subinterval of it —> Occurs(e, T) e.g., "patient2 broke a leg at 5pm" Processes hold over some subintervals of the interval they occur in —> Occuring(p, T) e.g., "patient3 is chasing the nurse"

5. Allen’s 13 Temporal Relations

6. The Temporal-Abstraction Task Input: time-stamped clinical data and relevant events (interventions) Output: interval-based abstractions Identifies past and present trends and states Supports decisions based on temporal patterns, such as: “modify therapy if the patient has a second episode of Grade II bone-marrow toxicity lasting more than 3 weeks” Focuses on interpretation, rather than on forecasting

7. Temporal Abstraction: The Bone-Marrow Transplantation Domain. 0 400 20010050 ²² 1000 2000 ² () ² ² ² 100K 150K () ² ² ² ² ² ² ² ² ² Granu- locyte counts ² ² ² ² Time (days) Platelet counts PAZ protocol M[0]M[1]M[2]M[3]M[1]M[0] BMT Expected CGVHD

8. Temporal Abstractions: A Graphical View

9. Uses of Temporal Abstractions Therapy planning and patient monitoring. E.g., the EON project (a modular architecture to support guideline-based care) Creating high-level summaries of time-oriented medical records Supporting an explanation module for a medical DSS Representing goals and policies of therapy plans and guidelines for quality assessment purposes (at runtime and retrospectively). E.g., the Asgaard project: Intentions of guideline designers with respect to both process and outcomes are captured as temporal patterns to be achieved or avoided. Visualization of time-oriented clinical data: the KNAVE project

10. The Temporal-Abstraction Ontology Events (insulin therapy) - part-of, is-a relations Parameters (hemoglobin values and abstractions) - abstracted-into, is-a relations Abstraction goals (therapy of diabetes patients) - is-a relations Interpretation contexts (effect of regular insulin) - subcontext, is-a relations Interpretation contexts are induced by other entities and can have any temporal relationship to the inducing entity

11. Temporal-Abstraction Output Types State abstractions (LOW, HIGH) Gradient abstractions (INC, DEC) Rate Abstractions (SLOW, FAST) Pattern Abstractions (CRESCENDO) - Linear patterns - Periodic patterns

12. Temperature Hemoglobin Level Linear Component Week 2Week 3Week 1 Anemia Fever Anemia Fever Anemia Fever Linear Component Periodic Pattern Abstraction of Periodic Temporal Patterns

13. Temporal-Abstraction Knowledge Types Structural (e.g., part-of, is-a relations) - mainly declarative/relational Classification (e.g., value ranges; patterns) - mainly functional Temporal-semantic (e.g., “concatenable” property) - mainly logical Temporal-dynamic (e.g., interpolation functions) - mainly probabilistic

14. The RÉSUMÉ System Architecture. Temporal-abstraction mechanisms Temporal fact base Events Contexts Abstracted intervals Primitive data Domain knowledge base Event ontology Parameter ontology Primitive data Events Context ontology External patient database ++ + +

15. Test Domains for the RÉSUMÉ System Medical domains: –Guideline-based care AIDS therapy Oncology –Monitoring of children’s growth –Therapy of insulin-dependent diabetes patients Non-medical domains: –Evaluation of traffic-controllers actions –summarization of meteorological data

16. Acquisition of Temporal-Abstraction Knowledge

17. Temporal Reasoning and Temporal Maintenance Temporal reasoning supports inference tasks involving time-oriented data; often connected with artificial-intelligence methods Temporal data maintenance deals with storage and retrieval of data that has multiple temporal dimensions; often connected with database systems Both require temporal data modelling

18. Tzolkin: A Temporal-Mediation Architecture or: Combining Temporal Reasoning and Temporal Maintenance

19. Knowledge-Based Visualization and Exploration of Time-Oriented Medical Data: Desiderata Interactive composition of (temporal-abstraction) queries Visualization of query results Exploration of multiple levels of temporal abstractions The semantics of the query, visualization and exploration operators should be domain independent, but should use the terms and relations specific to each (e.g., medical) domain

20. The KNAVE Project (Knowledge-Based Navigation of Abstractions for Visualization and Explanation) A conceptual and computational framework for temporal abstraction, visualization, and exploration Capitalizes on existing components (RÉSUMÉ, temporal database mediator, KA tool, domain-knowledge server) The exploration operators reuse (and are defined by) the domain’s temporal-abstraction ontology Introduces new graphical and computational modules

21. The KNAVE Architecture Temporal Mediator DB Visualization and exploration module Computational Module Graphical Interface Domain-knowledge Server End user Expert physician KA Tool Résumé Chronus Controller KB Ontology server

22. Beginning a Visualization Session: A Temporal-Abstraction Query

23. The Browsing and Exploration Interface

24. Semantic Exploration Operators Motion across semantic links in the domain’s knowledge base; in particular, relations (and their inverse) such as: - part-of - is-a - abstracted-from - subcontext Motion across abstraction types: state, gradient, rate, pattern Application of aggregation operators such as mean and distribution Dynamic change of temporal-granularity (e.g., from days to months) changes the display, using domain-specific aggregation knowledge Explanation by display of relevant knowledge, or through “What-if” queries, which allow hypothetical assertion or retraction of data or knowledge and examination of resultant patterns

25. An Abstracted-From Exploration Result

26. A Statistical-Query Example

27. Responding to an Explanation Query (“How”): A Bone-Marrow–toxicity Classification Table

28. The Preliminary Evaluation Study Developmental assessment of the prototype Seven users with varying medical/computer use backgrounds Each user given a 10 minute introduction to the KNAVE system A single electronic patient file constructed from several cases in the domains of AIDS and bone-marrow transplantation Each user asked to perform three tasks (a complex temporal query, a context-sensitive abstraction, and a statistical query) Qualitative impression and quantitative (time) measures noted

29. The Preliminary-Evaluation Results All users answered all queries within 3 minutes; 6 of 7 users completed all three tasks within 90 seconds All users expressed enthusiasm and found the interface useful Striking redundancy noted in use of interface: At least four different paths were found to the same answers, and five different patterns of use of the exploration operators Difficult to compare to manual tools, since these do not support any automated abstraction or explanation of such

30. KNAVE: Current State and Future Directions Basic prototype in Visual Basic; Java implementation under way Collaboration with an industrial company to create a web-based version Current research issues: –Implementation of temporal-granularity semantic zoom –Runtime linear and periodic pattern queries –Semantics and implementation of distributed What-If queries, which modify either the knowledge or the data at runtime and examine the effect of the result on the displayed patterns –Enhancement of RÉSUMÉ and the KA tool as needed, including integration with statistical tools –Future link to a text summarization module

31. Temporal-Abstraction and Visualization: Conclusions Temporal abstraction of time-oriented data can employ reusable domain-independent computational mechanisms that rely on access to a domain-specific temporal-abstraction ontology Temporal abstraction is useful for planning, monitoring, data summarization and visualization, explanation and critiquing Interactive query, visualization of, and exploration requires runtime access to the domain’s temporal-abstraction ontology The visualization and exploration semantics can be specific to the temporal-abstraction task, but not to the domain