1. Karin Schurink Peter Lucas Marc Bonten Stefan Visscher Incorporating Evaluation into the Design of a Decision-Support System UMC Utrecht Radboud University Nijmegen

2. Ventilator-Associated Pneumonia  VAP  Pneumonie acquise sous ventilation mécanique  Beademings-gerelateerde Longontsteking

3. Contents 1.Problem: incorporating evaluation into the design of a Decision-Support System (DSS) 2.Approach: A DSS for Ventilator-Associated Pneumonia (VAP): its underlying Bayesian-network and decision-theoretic model Clinical setting: advising regarding the diagnosis and treatment of VAP in the ICU Design of DSS guided by evaluation considerations

4. Introduction Ventilator-Associated Pneumonia (VAP)VAP Nosocomial infection on Intensive Care Units (ICU) Gold Standard = infectious-disease specialists Decision-Support System in uncertainty helps ­ diagnosing the patient ­ finding optimal treatment

5. Previous research in diagnosing and treating disorders in patients Evidence-based clinical guidelines Medical Decision-Support Systems Uncertainty: Bayesian network

6. General: Bayesian Network; 2 parts 1.qualitative = structure and relations between nodes (vars) 2.quantitative = conditional (estimated) probabilities example: P(pneumonia | body temperature  38.5 °C) A DSS for VAP: its underlying Bayesian-network and decision-theoretic model (1) Bayesian Network for VAP; 2 parts 1.diagnostic = clinical signs/ symptoms, duration of stay, mechanical ventilation 2.therapeutic = most effective combination of antibiotic treatment

7. Symptoms/ signs: temperature (fever) leukocytosis radiological signs sputum production Risk factors mechanical ventilation hospitalisation colonisation antibiotic use

8. Decision-theoretic model  Providing utilities for combinations of antibiotics, taking into account side effects financial costs antimicrobial spectrum A DSS for VAP: its underlying Bayesian-network and decision-theoretic model (2) utility : a quantitative measure of the strength of the expert’s preference in decision making

9. A DSS for VAP: its underlying Bayesian-network and decision-theoretic model (3) Global structure of the Bayesian Network

10. The problem: incorporating evaluation into the design of a DSS Has the DSS been properly evaluated? Diagnostic performance Usability + effect

11. Diagnostic performance of the DSS AUC 0.795 Dataset: 17.700 rows/ patient days in the ICU 157 VAP days experts vs. DSS

12. Design for an evaluation study of the DSS for VAP Known systematic effects/ biases in an evaluation study: 1.volunteer effect 2.Hawthorne effect 3.checklist effect

13. Resulting DSS for VAP (1) Components Clinical database User interface

14. Added facilities for evaluation 1.all users are asked to use the system 2.the doctor is asked to give his/ her opinion, choice of treatment and motivation for this choice concerning the patient 3.patient’s symptoms are presented: this form or checklist can be edited when necessarypatient’s symptoms are presented 4.at random, a management advice is presented: i. probability of VAP ii. optimal treatment for possible colonisation of the respiratory tract 5.only when an advice is presented, the doctor is asked again whether a VAP is suspected Resulting DSS for VAP (2)

15. all users (volunteer/ Hawthorne) (checklist)checklist

16. Screenshot

17. Conclusions & future work Medical Decision-Support Systems are meant to assist clinicians in the difficult process of medical management Known biases in evaluation studies were discussed More attention should be given to evaluation issues: only then it is possible to perform a reliable evaluation of a DSS In future we intent to perform the described evaluation study  did the clinician revise his/ her judgement, taking into account the system’s advice?  did the system’s advice influence the clinician’s diagnosis?

18. Contact: Stefan Visscher (S.Visscher@azu.nl)