Automated surveillance methods for notifiable diseases: theory, practice, and evaluation Matthew Samore, MD Utah Center of Excellence in Public Health Informatics Epicenter for Prevention of Healthcare Associated Infection Salt Lake VA Health Care System University of Utah

Objectives Define situation awareness and situation comprehension in the context of public health surveillance Application of an inference engine to electronic health records for case identification Evaluation of impact on public health surveillance work processes

Outline Surveillance theory Surveillance practice Surveillance evaluation

Surveillance Theory

Paradigms and Models The Panopticon –Archetype: “Total information awareness” Signal detection theory –Trade-off between hit rate and false alarm rate Situation awareness –Detection, diagnosis, prediction

What is situation awareness? Comprehension of a dynamic environment

Understanding comprehension A “phenomenon that emerges from an orchestra of cognitive processes” Perception –Surface level “Eventbase” –Semantic information extracted from perceptual input Situation model –Integration of semantic information with prior knowledge Durso, et. al: Handbook of Applied Cognition

A tragic error “On a beautiful night in October 1978, in the Chesapeake Bay, two vessels sighted one another visually and on radar. On one of them, the Cost Guard cutter training vessel Cuyahoga, the captain saw the other ship up ahead as a small object on the radar, and visually he saw two lights, indicating that it was proceeding in the same direction as his own ship. He thought it possibly was a fishing vessel. Since the two ships drew together so rapidly, the captain decided that it must be a very slow fishing boat that he was about to overtake.

At the last moment the captain of the Cuyahoga realized that in overtaking the supposed fishing boat, which he assumed was on a near-parallel course, he would cut off that boat’s ability to turn as both of them approached the Potomac River. So he ordered a turn to the port. This brought him directly in the path of the oncoming freighter, which hit the cutter. Eleven coastguardsmen perished.

The captain’s mental model

Cell phone use during driving How it may affect comprehension of the dynamic environment of the road –Fail to check rearview mirror (attention) –At 4 way stop, may fail integrate spatial and temporal information to determine who has right of way and should enter intersection first –May recognize that driver in oncoming vehicle is distracted but fail to infer the appropriate situation model, that the car will not stop

Situation awareness and public health Relevance –The health of a population is indeed dynamic –Time horizons cover a wide range The concept captures the data hungry nature of epidemiology and public health –Current health status –Current health care capacity –Early detection

Limitations: Vaguely expressed as goal of surveillance Decisions or actions not specified More importantly: –Description of the situation model is lacking

More effective use of the concept of situation awareness Probe deeper –Interpretation of current state –Projected states Measurement –Links theory to practice

Describing the current situation As given to computer scientist Yarden Livnat by Mary Hill, epidemiologist at Salt Lake Valley HD “There is significant flu in the valley” “Fever and coughs are rising” “12 people hospitalized throughout the valley” “20% are children” “Most cases are rapid A” “School absenteeism is high in middle/high schools” “Btw, the hunting season started yesterday”

Mental maps of epidemiologists GI outbreak scenario –Cluster of individuals with acute diarrhea who had eaten at a fast food restaurant Preliminary analysis: –Approach to problem space varies substantially across public health personnel

Surveillance Practice

Why syndromic surveillance (e.g., RODS) was perceived to be useful during Winter Olympic games (and other special events) At not very useful at other times

Thesis Utility of syndromic surveillance depends on situation model –Under normal conditions, its value is highly limited Current level of threat Suspicion of departure from normality

Public Health Practice Based Research

Core Principles Partnerships and synergies –Active participation of health departments at local and state levels –Public health leadership or co-leadership of projects –Vision of public health research laboratory Interdisciplinary collaboration at local and national levels

Utah Department of Health Office of Public Health Informatics Directed by Wu Xu, PhD Goals –Coordinate statewide eHealth initiatives –Coordinate integration of projects –Create a laboratory for applied public health research –Develop theory and methods that advance information and population science in the context of public health practice

How CoE Projects Support Public Health Case Identification Case Investigation & Management Analysis Actions based on Results Data Dissemination RT-CEND Linkage projects DSIDE INTERACT Outbreak response, advisories, other actions Rx Drug Deaths Readmission/Mortality

Leveraging the electronic health record Real-Time Clinical Electronic Notifiable Disease Surveillance Electronic health record-based syndromic surveillance –Text processing Leads –Utah Department of Health Lisa Wyman, Melissa Stevens Dimond, David Jackson, Corona Nigatuvai, Robert Rolfs –University of Utah Catherine Staes, Deepthi Rajeev –Intermountain Healthcare Scott Evans, Per Gesteland –VA Brett South, Matthew Samore, Adi Gundlapalli, Sylvain DeLisle

Data visualization and decision support Pathogen-specific surveillance (GermWatch) Heterogenic data visualization Public health decision support Interactive simulation Key investigators include: –Per Gesteland, Carrie Byington, Andy Pavia, Adi Gundlapalli, Yarden Livnat, Frank Drews, Laverne Snow, Chris Barrett, Stephen Eubank, Madhav Marathe, Jim Koopman, Yong Yang, Robert Rolfs

RT-CEND Project Health care system: –Rule-based detection of notifiable diseases –Message sending Electronic case transmission Health department –Message receiving –Integration into workflow at local and state level

Electronic Medical Record Laboratory Data Driver Decision Support Engine Medical Logic Modules Code Tables 23 4 Alert File 5 1

Alert File Time Driver Reportable Disease Monitor Electronic Medical Record ICP Daily Printout

Counts of notifiable diseases alerted

EMR Daily report NETSS faxed Manual entry Local Health DeptIntermountain HC State Health Dept CURRENT data flow or fax Alert file NETSS Manual entry

EMR Daily report NETSS faxed Simplified view of NEW data flow or fax Alert file Study/ NEDSS Data views, reports, extracts NETSS Manual entry HL7message Local Health Dept State Health Dept Intermountain HC

Electronic case transmission Work led by Deepthi Rajeev, Catherine Staes, Scott Evans Compiled required data fields Modeled the HL7 message structure for reporting from healthcare systems to local & state health departments –Evaluated existing messaging models, for instance PHIN implementation –Allows transmission of multiple lab tests based on one or multiple specimens in a single message Message structure implemented

Planned work on health care system side Automated HIPAA documentation Chart review validation

Message receipt & workflow integration Appropriate data flow between state and local health departments –Utah Department of Health Lisa Wyman, Corona Nigatuvai, David Jackson, Melissa Stevens Dimond, Robert Rolfs –Salt Lake Valley Health Department Heath Harris, Mary Hill, Ilene Risk –Davis County Health Department Brian Hatch, Nicole Stone

NEDSS implementation in Utah Current status: –Vendor: Collaborative Software Initiative Funded by grant from Novell Open source software model –Agile, rapid development method underway

Enhancing situation comprehension Leveraging the electronic health record to support public health investigation Start with possible event of public health interest –Use pre-defined and ad hoc queries on a health care system data warehouse to support evaluation –Collect additional epidemiologic data: how are cases linked? Who, what, where, when

How “good” are electronic health record-based case criteria? May be better or worse than conventional criteria Conventional case criteria constitute a reference standard but are never a true gold standard –Wit h respect to disease occurrence in a target population –Sensitivity is virtually always less than 100% because of incomplete clinical evaluation and testing Lack of true gold standard with respect to

Surveillance Evaluation

Conceptual framework Six core activities –Detection –Registration –Confirmation –Reporting –Analyses –Feedback. Public health action –Acute (epidemic-type) responses –Planned responses Mcnabb, et. al., BMC Public Health, 2002

Evaluation of public health information system implementation Formative evaluation –Semi-structured interviews Survey administration –Information technology acceptance model –Fit to workflow Observation of efficiency of work processes Measurement of timeliness, completeness, accuracy Assessment of the application development process

Time Line

Synthesis

Tier 1: Event Detection Trade-offs Reliability versus validity of case criteria “False alarm rate” versus “hit rate” Our project activities Text processing Rash syndromes Rule development Central nervous system infections

Tier 2: Cognitive processes Our related research activities Center of Excellence in Public Health Informatics Public health decision-making Simulation-based decision support

Tier 3: Broader Public Health Goals Our related research activities Informatics Information exchange Data linkage Epidemiology Epidemic models Valid ecologic inference

Syndromes Sensors Pathogen- specific Notifiable diseases Local/state health dept- centered activities Evaluation of existing systems (ELR, RODS, others) New system implementation BioSense NEDSS RT-CEND Tier 4: System Performance