Estimating the Expected Warning Time of Outbreak-Detection Algorithms Yanna Shen, Weng-Keen Wong, Gregory F. Cooper RODS Laboratory, Center of Biomedical Informatics, University of Pittsburgh 2005 Syndromic Surveillance

2005 Syndromic Surveillance Overview Objective Background Methods Experimental Results Conclusions Future Work 2005 Syndromic Surveillance

2005 Syndromic Surveillance Objective A new measure for evaluating alerting algorithms, which is called Expected Warning Time (EWT). It is a generalization of the standard AMOC curve. 2005 Syndromic Surveillance

2005 Syndromic Surveillance Why Useful? Can compare expected clinician detection time to expected computer-based algorithm detection time Can provide a promising new approach for optimizing and comparing outbreak detection algorithms 2005 Syndromic Surveillance

2005 Syndromic Surveillance Background hit Incubation Time hit hit hit … … … False Alerts (in red) Time Computers raise alert Clinicians detect outbreak Last outbreak case appears Computer Detection Time Warning Time Release occurs Maximum meaningful detection time Maximum meaningful WT 2005 Syndromic Surveillance

2005 Syndromic Surveillance Model A simple model of clinician outbreak detection Assumes that People with disease D are diagnosed independently of each other. The probability of a person with disease D being diagnosed is constant (p). 2005 Syndromic Surveillance

2005 Syndromic Surveillance Equation Definitions: p – probability that a person with D is diagnosed as having D upon presentation with that disease time(i) – maps patient case i to the time at which that patient presented with D to clinicians M – total # of patient cases with D t – time at which the alerting score first exceeds a given threshold 2005 Syndromic Surveillance

Equation x + x WT if clinicians never detect the outbreak Probability that clinicians will detect the outbreak on the ith case WT if clinicians first detect the outbreak on the ith case Probability that clinicians will never detect the outbreak 2005 Syndromic Surveillance

2005 Syndromic Surveillance Experiment Setup Apply PANDA to simulated cases of inhalational anthrax For various value of p, derive EWT for PANDA 2005 Syndromic Surveillance

2005 Syndromic Surveillance PANDA An outbreak detection system Uses causal Bayesian networks to model spatio-temporal patterns of a non-contagious disease in a population (Cooper, 2004) Contains a model to detect inhalational anthrax 2005 Syndromic Surveillance

2005 Syndromic Surveillance BARD BARD simulator produces the simulated cases of anthrax. It models the effects of an outdoor airborne anthrax release using the Gaussian plume model of atmospheric dispersion and a model of inhalational anthrax (Hogan, 2004). 2005 Syndromic Surveillance

Performance of Clinicians p – Clinician detection proficiency P(CD) – Probability that clinicians will detect the outbreak at all ECDT – Expected clinician detection time given that clinicians detect the outbreak p P(CD) ECDT (hours) 0.001 0.992 167.9 0.005 0.999 104.1 0.01 0.9999 88.6 0.05 0.99999 64.5 1 2005 Syndromic Surveillance

2005 Syndromic Surveillance Experimental Results ? p increases, EWT decreases p = 1, EWT = 0 2005 Syndromic Surveillance

Clinician-detection-proficiency (p) Expected Warning Time (EWT) Experimental Results False alert rate = 1 per month Clinician-detection-proficiency (p) Expected Warning Time (EWT) 0.001 76 hours 0.005 13 hours 0.01 5 hours 0.05 34 minutes 1 If and false alert rate = 1 per month, then EWT minutes. 2005 Syndromic Surveillance

2005 Syndromic Surveillance Conclusions The Expected Warning Time (EWT) is a useful concept for evaluating outbreak-detection algorithms. We illustrated the general idea of EWT using a simple model of clinician detection and simulated cases of inhalational anthrax. Our example analysis suggests that PANDA is most helpful when clinicians’ detection proficiency < 5%. 2005 Syndromic Surveillance

2005 Syndromic Surveillance Future Work Extend the model: Instead of a constant (p), use the function p(t), where t is time Develop and apply more disease-specific models of clinician detection (please see the poster by Christina Adamou) 2005 Syndromic Surveillance

2005 Syndromic Surveillance Acknowledgements This research was supported by grants from the National Science Foundation (IIS-0325581), the Department of Homeland Security (F30602-01-2-0550), and the Pennsylvania Department of Health (ME-01-737). We thank members of the Bayesian Biosurveillance Project for helpful comments. 2005 Syndromic Surveillance