“To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Biosurveillance Information Exchange Working Group Reflections from San Diego County Jeffrey Johnson, MPH San Diego County Health & Human Services Agency 2/23/06

SAN DIEGO COUNTY Nearly 3 million population International border Large military presence Biotechnology Hub 21 Emergency Departments

Early Event Detection in San Diego Evolving effort since pre - 9/11 Data sources: ER Visits, Paramedic transports, 911 calls, school surveillance, OTC sales Systems: Local SAS/Minitab system, ESSENCE, and BioSense Statistical Methods: Descriptive, time series, CUSUM, EWMA, process control methods (P&U Charts) Multiple syndromes Visualization and alerting Incident Characterization Follow-up to signals County of San Diego Health & Human Services Agency

If We Ignore A Signal…… We take no action or follow-up Save staff resources Avoid bothering hospital staff yet again Another data source may signal “The Feds may pick it up” Might lose an earlier start to a response We might be dead wrong to ignore

If We Do Not Ignore a Signal…… Will it be another “false alarm” May detect an event earlier Earlier response Continued interaction with the medical community Gain experience with follow-up Increased situational awareness

Characterization of Detections Detection Method Syndrome group % Admitted Deaths? Geographic cluster? Prior day’s level? Recent level? Age groups? Severe syndrome? Detections in other data sources? Other epidemiological intelligence? Other diagnostic information Follow-up? Action or No Action or Watch

Detection Follow-up with Medical Community What is the final diagnosis of Patients A, B, C? Is there a common pattern among admitted patients? Did any have lab test results that might suggest a larger event? Among patients with a common zip code, was there a shared living setting or common exposure? Can we send someone out to review medical charts? What is your facility’s assessment of the situation? County of San Diego Health & Human Services Agency

County of San Diego Health & Human Services Agency Routine Surveillance Activities Rule out system error YES NO Preliminary evaluation Describe initial results True Positive Aberration detected Potential false positive YES NO “False Positive” Inform key divisional staff Intensive monitoring & surveillance Evaluate other data sources Cluster check VERIFY NOTIFY Inform key departmental staff IDENTIFY Ignore?

GI Syndrome Over Time (10/31/04 – 8/24/05) ED 911 Paramedic Runs

The Significant Aspects of Syndromic Surveillance Statistical Significance Public Health Significance Significant Event Significant Public Awareness Significant Biological Agent Detection

HAZMAT FLAG – 12/04/2004 County of San Diego Health & Human Services Agency Statistical significance vs. public health significance

County of San Diego Health & Human Services Agency Statistical significance vs. public health significance

County of San Diego Health & Human Services Agency

Syndromic Surveillance for Natural Disasters San Diego Wild Fires, 2003 Significant event with statistically significance outcomes San Diego County

Syndromic surveillance for natural disasters Significant event with statistically significance outcomes

“The Clinton Effect” September 4, 2004 While spikes in both datasets are apparent, normalized counts show a relatively larger increase in ED visits on Sept. 6, Significant Public Awareness

7/7/05 London Bombings San Diego County Paramedic Transports for “Chest Pain” Significant Public Awareness

Application of Syndromic Surveillance Agent: Syndrome categories Specific word search in CC or DX fields Sensor site: Zip codes, population (schools) Date: Temporal based surveillance New pre-detection baselines Biowatch BioWatch Detection Tells us agent, sensor site and date Plume plot may help us narrow surveillance on a geographic area Significant BT Agent Detection

Anatomy of a Detection (a case example)

Daily Report Attached Table Feb 5, Call Data

911 Call Center - GI Syndrome Signal

Line listing for review Non-specific call complaints

21 Signals since 07/01/03 Various statistical signals The count for the signals include a consistent range 911 Call Center - GI Syndrome Signal

What did we do? Magnitude of cases Which method(s) signaled? Check the other call centers Check the other data sources (ED data, EMS transports) Review the line listing Our conclusion….. … vs mean of 7.8 …... CUSUM (2), P-Chart, U-Chart …… No signals …… No Signals ……. No apparent pattern >>>>> Super Bowl Sunday Fewer trauma calls Smaller denominator (P-Chart) Traditional increase in GI on this day Watch next day’s results

Case Example #2 Hospital 9 ED Data Respiratory Syndrome

Hospital 9 - Daily Results Table

Hospital 9 Respiratory Syndrome 01/01/ /03/06 Many signals…. So what’s the context? Do we ever ignore the signals?

Hospital 9 Respiratory Syndrome 24 signals over a 37 day period Count range: 11 – 34 Over time an increasing mean

Hospital 9 Influenza-like-illness (ILI) Syndrome “ILI syndrome” has greater syndrome specificity than “Respiratory” syndrome” 16 signals over a 37 day period Greater Syndrome Specificity……

S$gn&ls Happen! Make sure you see flames before yelling “Fire” CUSUM 2 & 3 STD may be too sensitive We lose precision with non-specific syndromes Everyone wants to know what’s going on all the time Increasing focus on situational awareness Further evaluation and testing required What We Have Learned

Event or Technology Trigger Peak of Inflated Expectations Trough of Disillusionment Slope of Enlightenment Plateau of Productivity Hype Cycle of Emerging “Syndromic Surveillance” Technologies Adapted from the Gartner Hype Cycle Too many signals?, IT Costs, poor syndrome specificity, evaluation results Prioritized data sets, protocols in place, The “magic bullet” 9/11, Anthrax attacks Dual use, situational awareness, appropriate signals

More work in all areas of syndromic surveillance is needed Knowledge requires responsibility The enemy is studying our efforts Current/future funding levels require reliability, efficiency and sustainability of systems and approaches The Future: Neural networks and Artificial Intelligence (AI)? Are we ready? Considerations

Contact Information Jeffrey Johnson Thank You