1. “To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Biosurveillance Information Exchange Working Group Reflections from San Diego County Jeffrey Johnson, MPH San Diego County Health & Human Services Agency 2/23/06

2. SAN DIEGO COUNTY Nearly 3 million population International border Large military presence Biotechnology Hub 21 Emergency Departments

3. Early Event Detection in San Diego Evolving effort since pre - 9/11 Data sources: ER Visits, Paramedic transports, 911 calls, school surveillance, OTC sales Systems: Local SAS/Minitab system, ESSENCE, and BioSense Statistical Methods: Descriptive, time series, CUSUM, EWMA, process control methods (P&U Charts) Multiple syndromes Visualization and alerting Incident Characterization Follow-up to signals County of San Diego Health & Human Services Agency

4. If We Ignore A Signal…… We take no action or follow-up Save staff resources Avoid bothering hospital staff yet again Another data source may signal “The Feds may pick it up” Might lose an earlier start to a response We might be dead wrong to ignore

5. If We Do Not Ignore a Signal…… Will it be another “false alarm” May detect an event earlier Earlier response Continued interaction with the medical community Gain experience with follow-up Increased situational awareness

6. Characterization of Detections Detection Method Syndrome group % Admitted Deaths? Geographic cluster? Prior day’s level? Recent level? Age groups? Severe syndrome? Detections in other data sources? Other epidemiological intelligence? Other diagnostic information Follow-up? Action or No Action or Watch

7. Detection Follow-up with Medical Community What is the final diagnosis of Patients A, B, C? Is there a common pattern among admitted patients? Did any have lab test results that might suggest a larger event? Among patients with a common zip code, was there a shared living setting or common exposure? Can we send someone out to review medical charts? What is your facility’s assessment of the situation? County of San Diego Health & Human Services Agency

8. County of San Diego Health & Human Services Agency Routine Surveillance Activities Rule out system error YES NO Preliminary evaluation Describe initial results True Positive Aberration detected Potential false positive YES NO “False Positive” Inform key divisional staff Intensive monitoring & surveillance Evaluate other data sources Cluster check VERIFY NOTIFY Inform key departmental staff IDENTIFY Ignore?

9. GI Syndrome Over Time (10/31/04 – 8/24/05) ED 911 Paramedic Runs

10. The Significant Aspects of Syndromic Surveillance Statistical Significance Public Health Significance Significant Event Significant Public Awareness Significant Biological Agent Detection

11. HAZMAT FLAG – 12/04/2004 County of San Diego Health & Human Services Agency Statistical significance vs. public health significance

12. County of San Diego Health & Human Services Agency Statistical significance vs. public health significance

13. County of San Diego Health & Human Services Agency

14. Syndromic Surveillance for Natural Disasters San Diego Wild Fires, 2003 Significant event with statistically significance outcomes San Diego County

15. Syndromic surveillance for natural disasters Significant event with statistically significance outcomes

16. “The Clinton Effect” September 4, 2004 While spikes in both datasets are apparent, normalized counts show a relatively larger increase in ED visits on Sept. 6, 2004. Significant Public Awareness

17. 7/7/05 London Bombings San Diego County Paramedic Transports for “Chest Pain” Significant Public Awareness

18. Application of Syndromic Surveillance Agent: Syndrome categories Specific word search in CC or DX fields Sensor site: Zip codes, population (schools) Date: Temporal based surveillance New pre-detection baselines Biowatch BioWatch Detection Tells us agent, sensor site and date Plume plot may help us narrow surveillance on a geographic area Significant BT Agent Detection

19. Anatomy of a Detection (a case example)

20. Daily Email Report Attached Table Feb 5, 2006 911 Call Data

21. 911 Call Center - GI Syndrome Signal

23. Line listing for review Non-specific call complaints

24. 21 Signals since 07/01/03 Various statistical signals The count for the signals include a consistent range 911 Call Center - GI Syndrome Signal

25. What did we do? Magnitude of cases Which method(s) signaled? Check the other call centers Check the other data sources (ED data, EMS transports) Review the line listing Our conclusion….. …... 14 vs mean of 7.8 …... CUSUM (2), P-Chart, U-Chart …… No signals …… No Signals ……. No apparent pattern >>>>> Super Bowl Sunday Fewer trauma calls Smaller denominator (P-Chart) Traditional increase in GI on this day Watch next day’s results

26. Case Example #2 Hospital 9 ED Data Respiratory Syndrome

27. Hospital 9 - Daily Results Table

30. Hospital 9 Respiratory Syndrome 01/01/04 - 02/03/06 Many signals…. So what’s the context? Do we ever ignore the signals?

31. Hospital 9 Respiratory Syndrome 24 signals over a 37 day period Count range: 11 – 34 Over time an increasing mean

32. Hospital 9 Influenza-like-illness (ILI) Syndrome “ILI syndrome” has greater syndrome specificity than “Respiratory” syndrome” 16 signals over a 37 day period Greater Syndrome Specificity……

33. S$gn&ls Happen! Make sure you see flames before yelling “Fire” CUSUM 2 & 3 STD may be too sensitive We lose precision with non-specific syndromes Everyone wants to know what’s going on all the time Increasing focus on situational awareness Further evaluation and testing required What We Have Learned

34. Event or Technology Trigger Peak of Inflated Expectations Trough of Disillusionment Slope of Enlightenment Plateau of Productivity Hype Cycle of Emerging “Syndromic Surveillance” Technologies Adapted from the Gartner Hype Cycle Too many signals?, IT Costs, poor syndrome specificity, evaluation results Prioritized data sets, protocols in place, The “magic bullet” 9/11, Anthrax attacks Dual use, situational awareness, appropriate signals

35. More work in all areas of syndromic surveillance is needed Knowledge requires responsibility The enemy is studying our efforts Current/future funding levels require reliability, efficiency and sustainability of systems and approaches The Future: Neural networks and Artificial Intelligence (AI)? Are we ready? Considerations

36. Contact Information Jeffrey Johnson 619-531-4945 jeffrey.johnson@sdcounty.ca.gov Thank You