Detection and Analysis Perspectives of Both Data Monitors and Algorithm Developers

Data Monitors Turbulence Varied skill sets Turnover Training Varied skill sets Limited understanding of algorithms Overworked and few Feedback to providers critical

Developers Interaction and feedback from users is important Iteration Critical for developers to know users problems Limited understanding of public health operations Expense of false positive to users

Available Detection Methods BioSense, Essence, RODS, Red Bat… Cusum, Smart Scores, RLS, EWMA, Wavelet, … Spatial scan statistics, SatScan, zipcode Multiple detection algorithms – how many are flagging? Multiple detections corroborate problem Real-time vs. batch?

Real-time Detection Data is unsettled when it arrives Pressure for real-time may exacerbate existing problems Is it sustainable? Who will monitor it? How valuable is it? If not everyday – can we do it in a crisis?

Weaknesses of Detection Algorithms Cusum, EWMA – most widely used Control chart has many assumptions Normally distributed Stationary assumption Are assumptions being met? Method must match data Getting false alarms that exceed rate that would be expected may signal disconnect between data and algorithm

Challenges (to name a few) Syndrome categorization More? Fewer? Subsetting? Statistical significance does not equal public health significance “False” alarms

Issues for Users False positives Disconnect between developers and users Difficulty evaluating what is a real alarm Data quality issues What user needs are being supported? What are other uses for data? Overall evaluation of syndromic surveillance utility

Possible Approaches to Improving Detection Explanation to user of why alarming Pop-up? Text-strings from physicians Phased alerting system Yelling, Anomalies, Alert Improving alert qualification (RODS in Ohio)

Possible Approaches to Improving Detection (continued) Better pre-processing of data Better post-processing of data De-duplication of data, etc. Failure analysis of false alarms What are major contributors?

Needs Refined data Improved algorithms Human expert required to make determination Domain knowledge Local knowledge Statistical analysis is only a tool Good relationships among users Federal, state, local, facility levels

Summary Systems: Alarm too often Users: May become discouraged Statistical challenges Detection methods up to it? Data problems? Solutions: Better data, better processing, more money