Scenario-Based Evaluation of Cluster Detection and Tracking Capability

Slides:



Advertisements
Similar presentations
Data Mining Methodology 1. Why have a Methodology  Don’t want to learn things that aren’t true May not represent any underlying reality ○ Spurious correlation.
Advertisements

 2005 Carnegie Mellon University A Bayesian Scan Statistic for Spatial Cluster Detection Daniel B. Neill 1 Andrew W. Moore 1 Gregory F. Cooper 2 1 Carnegie.
Sampling Distributions (§ )
Introduction to Statistics
Bayesian Biosurveillance Gregory F. Cooper Center for Biomedical Informatics University of Pittsburgh The research described in this.
Empirical/Asymptotic P-values for Monte Carlo-Based Hypothesis Testing: an Application to Cluster Detection Using the Scan Statistic Allyson Abrams, Martin.
 2004 University of Pittsburgh Bayesian Biosurveillance Using Multiple Data Streams Weng-Keen Wong, Greg Cooper, Denver Dash *, John Levander, John Dowling,
Evaluating Hypotheses
Conclusions On our large scale anthrax attack simulations, being able to infer the work zip appears to improve detection time over just using the home.
Spatiotemporal Cluster Detection in ESSENCE Biosurveillance Systems Panelist: Howard Burkom National Security Technology Department, John Hopkins University.
Inferences About Process Quality
SPONSOR JAMES C. BENNEYAN DEVELOPMENT OF A PRESCRIPTION DRUG SURVEILLANCE SYSTEM TEAM MEMBERS Jeffrey Mason Dan Mitus Jenna Eickhoff Benjamin Harris.
A Wavelet-based Anomaly Detector for Disease Outbreaks Thomas Lotze Galit Shmueli University of Maryland College Park Sean Murphy Howard Burkom Johns Hopkins.
PARAMETRIC STATISTICAL INFERENCE
1 ESSENCE: Biosurveillance in Support of the DoD Health Mission.
The Practice of Statistics, 5th Edition Starnes, Tabor, Yates, Moore Bedford Freeman Worth Publishers CHAPTER 10 Comparing Two Populations or Groups 10.2.
The Practice of Statistics, 5th Edition Starnes, Tabor, Yates, Moore Bedford Freeman Worth Publishers CHAPTER 10 Comparing Two Populations or Groups 10.1.
The Practice of Statistics, 5th Edition Starnes, Tabor, Yates, Moore Bedford Freeman Worth Publishers CHAPTER 10 Comparing Two Populations or Groups 10.1.
Practical Aspects of Alerting Algorithms in Biosurveillance Howard S. Burkom The Johns Hopkins University Applied Physics Laboratory National Security.
~PPT Howard Burkom 1, PhD Yevgeniy Elbert 2, MSc LTC Julie Pavlin 2, MD MPH Christina Polyak 2, MPH 1 The Johns Hopkins University Applied Physics.
Learning Objectives After this section, you should be able to: The Practice of Statistics, 5 th Edition1 DESCRIBE the shape, center, and spread of the.
No More Black Box: Methods for visualizing and understanding your data for useful analysis Howard Burkom National Security Technology Department Johns.
Michigan Disease Surveillance System Syndromic Surveillance Project January 2005.
Bayesian Biosurveillance of Disease Outbreaks RODS Laboratory Center for Biomedical Informatics University of Pittsburgh Gregory F. Cooper, Denver H.
The Practice of Statistics, 5th Edition Starnes, Tabor, Yates, Moore Bedford Freeman Worth Publishers CHAPTER 10 Comparing Two Populations or Groups 10.1.
Correlation of National Influenza Surveillance Data to the Local Experience Kate Goodin, MPH Florida Department of Health Bureau of Epidemiology 6 th Annual.
Overview G. Jogesh Babu. R Programming environment Introduction to R programming language R is an integrated suite of software facilities for data manipulation,
Canadian Bioinformatics Workshops
Marshall University School of Medicine Department of Biochemistry and Microbiology BMS 617 Lecture 15: Sample size and Power Marshall University Genomics.
Towards Improved Sensitivity, Specificity, and Timeliness of Syndromic Surveillance Systems Anna L. Buczak, PhD, Linda J. Moniz, PhD, Joseph Lombardo,
The Practice of Statistics, 5 th Edition1 Check your pulse! Count your pulse for 15 seconds. Multiply by 4 to get your pulse rate for a minute. Write that.
The Practice of Statistics, 5th Edition Starnes, Tabor, Yates, Moore Bedford Freeman Worth Publishers CHAPTER 10 Comparing Two Populations or Groups 10.2.
Where Are You? Children Adults.
CHAPTER 10 Comparing Two Populations or Groups
Learning Objectives : After completing this lesson, you should be able to: Describe key data collection methods Know key definitions: Population vs. Sample.
7. Performance Measurement
Advanced Quantitative Techniques
Descriptive study design
CHAPTER 10 Comparing Two Populations or Groups
Relative Values.
CHAPTER 10 Comparing Two Populations or Groups
Statistical Data Analysis
Bayesian Biosurveillance of Disease Outbreaks
CHAPTER 10 Comparing Two Populations or Groups
ACIP Feb , 2007 Guillain-Barré Syndrome (GBS) Among Recipients of Meningococcal Conjugate Vaccine (MCV4,Menactra®) Update Oct Jan Robert.
Chapter 6 Hypothesis tests.
Attention Narrows Position Tuning of Population Responses in V1
Descriptive and inferential statistics. Confidence interval
Section 7.7 Introduction to Inference
CHAPTER 10 Comparing Two Populations or Groups
Daniela Stan Raicu School of CTI, DePaul University
CHAPTER 10 Comparing Two Populations or Groups
Sampling Distributions
Statistical Data Analysis
CHAPTER 10 Comparing Two Populations or Groups
EQT 272 PROBABILITY AND STATISTICS ROHANA BINTI ABDUL HAMID
CHAPTER 10 Comparing Two Populations or Groups
CHAPTER 10 Comparing Two Populations or Groups
CHAPTER 10 Comparing Two Populations or Groups
Sampling Distributions (§ )
CHAPTER 10 Comparing Two Populations or Groups
CHAPTER 10 Comparing Two Populations or Groups
CHAPTER 10 Comparing Two Populations or Groups
Advanced Algebra Unit 1 Vocabulary
Sampling Distributions
CHAPTER 10 Comparing Two Populations or Groups
CHAPTER 10 Comparing Two Populations or Groups
CHAPTER 10 Comparing Two Populations or Groups
CHAPTER 10 Comparing Two Populations or Groups
Chapter 5: Sampling Distributions
Presentation transcript:

Scenario-Based Evaluation of Cluster Detection and Tracking Capability Howard Burkom1, Jian Xing2, Linda Moniz1, Jerry Tokars2 1 Johns Hopkins University Applied Physics Laboratory 2 Centers for Disease Control and Prevention 6th Annual Public Health Information Network Conference Section F7: Evaluation of Surveillance Systems Atlanta, GA August 27, 2008 Copyright 2006 The Johns Hopkins University Applied Physics Laboratory. All Rights Reserved. Cite collaboration of Linda, Jim & Jerry at CDC BioSense Bridging gap between statistical research and PH practice—data dependent Addressing challenge others have posed, including at this conference. This presentation was supported by Grant R01 PH000024-01 from CDC. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of CDC.

Presentation Outline BioSense Data Environment: DoD Outpatient Clinic Visit Records Cluster Detection Methodology Synthetic Signal Approach Study Design Study Results Conclusions Presentation outline: note importance of data source. Conclusions don’t necessarily carry over other data sources and syndrome filterings, & we’ve developed methods for other data types.

Data Source: Daily Outpatient Visits Counts by Syndrome Data provided through CDC BioSense program under R01 grant PH000024-01. 3 years’ worth of military clinic data from Texas, with some clinics in Oklahoma and Louisiana; most patients in LA and OK clinics come from Texas. 32 facility zip codes, 1233 residence zips. Dataset includes active duty personnel, dependents, and retirees. Zip code level data: individual patient records contain 2 zip codes: treatment facility & residence. No other spatial information available. Can look at spatial clusters by either one. By facility, surrogate for work, by residence, surrogate for home.

Data Highly Skewed Toward a Few Locations A few clinics completely dominate the case counts. A few home zips dominate the case counts. 1099 For both types of spatial organization, recorded zips highly concentrated in relatively few locs. Facililty: 20 of 32 zips have median daily count 0, just 3 with > 100. Residence: 1099 of 1233 have median daily count 0. Records not spatially distributed like census pop—Jim’s presentation last yr.

Method of Scan Statistics form cylinders: Bases are circles about each centroid in region A, height is time Surveillance Region centroids of data collection regions calculate statistic: for event count in each cylinder relative to entire region, within space & time limits x x x x x most significant clusters: regions whose centroids form base of cylinder with maximum statistic x x Summary of cluster detection method—scan statistics popularized by SaTScan software. x but how unusual is it? Repeat procedure with Monte Carlo trials, compare max statistic to set of maxima of these M. Kulldorff’s SaTScan, downloadable at http://www.satscan.org x x x x

Implementations of Scan Statistics Separate codes in C and Matlab, precisely cross-checked Enable testing of spatial distribution estimation methods Significance testing: use Gumbel (extreme value) distribution to reduce the number of required Monte Carlo runs. method of Abrams A, Kulldorff M, Kleinman K, 2006. “Empirical/Asymptotic P-values for Monte Carlo-Based Hypothesis Testing: an Application to Cluster Detection Using the Scan Statistic”, Adv. in Disease Surv. Vol. 1. They showed that thresholds derived from the Gumbel distribution with 100 replicates may be used in place of rank-based p-values using 999 Monte Carlo replicates Added capability to inject parametric, stochastic signals to background data in order to evaluate detection performance

Details of Scan Statistic Study Implementation Estimated spatial distribution inferred from a sliding baseline 56 days with a 2-day separation buffer from test day(s) 2 estimation methods in this study for individual zip-code totals a. Flat baseline average b. Averages stratified by day-of-week (DOW), counting holidays as Sundays. Only single-day clusters examined in this study Output of the scan statistic: clusters whose Gumbel p-values are below chosen threshold. We ran the scan statistics on the entire 3 years of data to get the background daily rate of significant cluster determination. Injected sets of stochastic signals to measure detection performance

Algorithm Evaluation: Authentic vs Synthetic Data Authentic data Advantage of representation of true background noise Data very difficult to obtain and publish Known outbreak effects, especially with spatial distribution, very rare Synthetic data Fully understood noise field, but must show epidemiological relevance Onset and strength of signal known precisely Can know that background alarms are false alarms Semi-synthetic approach Injecting realistic signals into authentic background data Authentic data for algorithm evaluation are hard to get, sometimes harder to publish This difficulty has led to frequent discussion of whether to use relatively small sets of unlabelled data, or simulate noise background and/or signal In authentic data, almost never know exact date of onset in a population sense—even individual onset dates can be questionable ( been involved in expert-directed studies—never satisfactory because info required by experts so difficult to get )

Repeated Trials with Synthetic Signals Add a fixed number of inject cases to selected daily zip code counts using chosen case spread geometry study added 30-100 records region-wide over ~8 days Run scan statistic detection on resultant dataset Criterion for detection: a statistically significant cluster containing a zip code with injected cases. Subsequent trials: advance start day by 8 days, repeat the process, for about 130 trials using ~3 years’ data To evenly sample day-of-week, seasonal effects

Idealized Signal Shape: Sartwell’s Lognormal Model Incubation periods may be described by a lognormal distribution, with parameters depending on: disease type route of infection Individual factors: dosage susceptibility (vaccination status, genetic factors) Epicurve: plot of number of symptomatic cases by day Canonical idea of a bioterrorist attack is a localized, point-source outbreak, such as the 1979 accident at Sverdlovsk where weaponized anthrax spores were released in aerosol form, an unknown number infected, and about 70 died Magenta dotted curve shows actual epicurve we constructed from plot in 1992 Meselson paper We’ve taken data such as this to calculate zeta and sigma for disease-specific lognormal dist. Can then plot the “maximum likelihood epicurve” Modal day is exp(zeta + 2*sigma); in constructing a test signal, we set the modal number of cases to a multiple of the estimated standard deviation of the time series of interest, then divide by the modal probability to get the total number infected, and we add the resulting counts to the authentic data Sartwell, PE. The distribution of incubation periods of infectious disease. Am J Hyg 1950; 51:310:318

“maximum likelihood” epicurve Each symptomatic case a random draw Form Stochastic Epicurve, then Distribute Each Day’s Cases by Subregion “maximum likelihood” epicurve Each symptomatic case a random draw For each trial, compute histogram of cases by day with random draws, then distribute each day’s cases according to chosen geometry of spread However, we don’t assume the maximum likelihood epicurve in our simulation; we form a stochastic signal using the calculated N, zeta, and sigma For each of the N simulated cases, we take a random draw from the lognormal with params, zeta & sigma, just as each individual’s incubation period could be seen as a random draw from distributions of dosage and susceptibility Each trial is then a set of N such random draws, giving a large set of random signals These signals are what we add to the noise background of authentic data Rolka H, Burkom H, Cooper GF, Kulldorff M, Madigan D, Wong W-K, Issues in applied statistics for public health, Bioterrorism surveillance using multiple data streams: research needs, Statistics in Medicine (2007), Vol. 26, 8:1834-1856

Inject Geometries Circular (resp) Hourglass (resp) Wedge (resp)

Sample Distribution of Background and Injected Visit Records Typical data scenario: effects of an outbreak—50 additional cases spread out over 9 days Note regionwide data series and epicurve. Black for background counts, red for injects.

Results Chart for Respiratory Syndrome Note drop in sensitivity from residence to facility level analysis High sensitivity in rural regions, dropping as background gets more noisy

Results Chart for Rash Syndrome Sensitivity to weaker signals for more specific rash syndrome, sparser background At level of residence zip codes, sensitivity increases as signal moves away from noisy urban center

Outbreak Signal Sizes Required for Detection using only Time Series Multiple testing problems may be severe at facility or local levels—Does distributed investigation capability exist?

Comparison of Estimation Methods by threshold p-value Resp Syndrome, by residence zip, “suburban” center, PD by Day 4 flat baseline avg., 100 injects wkday/wkend avg., 100 injects flat baseline avg., no injects wkday/wkend avg., no injects

Comparison of Estimation Methods by empirical background cluster rate 50 total injects flat baseline avg wkday/wkend avg. 100 total injects flat baseline avg wkday/wkend avg.

Utility of Signal Formation/Injection Procedure Scenario-based comparison, tuning of cluster detection methods Quantifying alerting capability Use case-specific: how large a signal is detectable? How to set thresholds, estimate spatial distribution for required sensitivity Clarifying sensitivity by background density in highly skewed distributions Urban, Suburban, Rural Helping to evaluate syndrome groupings for a clinical data source by quantifying spatial noise effects on detection capability

Conclusions 1: DoD outpatient visit dataset Event-based scenarios involving military bases: use facility-level cluster detection More representative of work location Maximum general sensitivity: use residence level zips To estimate spatial distribution must account for day-of-week and seasonal effects to avoid spurious clustering but can lose specificity by overfitting model for specific syndrome grouping

Conclusions 2: DoD outpatient visit dataset BioSense rash category allowed detection of small clusters ( <50 additional region-wide cases/week ) at low background cluster rates (1-2 per month) BioSense resp-related record grouping too noisy for sensitivity to modest-sized clusters ( ~100 cases) at low alert rates consider more specific filtering (subsyndrome) for this dataset

Conclusions 3: DoD outpatient visit dataset Findings emphasize importance of: ability to classify records by patient residence or work location, to capture location of exposure syndromic record filtering for effective signal-to-noise discrimination Developing rapid capability to Display clinics, chief complaints, ages, other key record fields for all cases in suspected cluster Identifying potential linkages among these records

Acknowledgements Data were provided through the CDC BioSense program under R01 grant PH000024-01. Jim Edgerton, APL and Mike Leuze, SAIC did principal development of the MATLAB and C implementations of the scan statistic algorithms; Jim Edgerton developed the Gumbel distribution adaptation in the MATLAB implementation. This presentation was supported by Grant R01 PH000024-01 from CDC. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of CDC.