Evaluation of a surveillance system Pawel Stefanoff

Evaluation of a surveillance system: a cyclic process Gaps between needs and capacities Implementation Monitoring and evaluation Planning for improvements

WHAT do we want to assess? Technique Disease specific surveillance Surveillance system Complexity The way of defining the problem methods solution.

To assess the capacity of the system to meet its purpose and objectives In order to –improve its operation –modify it to address: priorities epidemiology –optimize the available resources Why evaluate?

Description of the needs –Country context –Disease patterns –Target audience Description of the surveillance system –Objectives –Process –Structure: resources, system Analysis of attributes Analysis of capacities /needs Implementation of prioritized modifications Plan of evaluation

Description of the needs –Country context –Disease patterns –Target audience Description of the surveillance system –Objectives –Process –Structure: resources, system Analysis of attributes Analysis of capacities /needs Implementation of prioritized modifications Plan of evaluation

Description of surveillance needs Description of the country context –Population –Health system –Political structure and administration Description of disease patterns: –Disease characteristics (epidemic potential) –Risk factors (country-specific?)

Whats the target audience of surveillance? Public Health professionals? Clinicians / Microbiologists / Control of Infection staff? Environmental Health professionals? Health service managers? Health educators / teachers? Government / Politicians? Public?

Description of the needs –Country context –Disease patterns –Target audience Description of the surveillance system –Objectives –Process –Structure: resources, system Analysis of attributes Analysis of capacities /needs Implementation of prioritized modifications Plan of evaluation

Description of the surveillance system The description of the surveillance system is a scientific process –Poor methods will generate unreliable results When describing a surveillance system: –Report your methods in the methods section –Write the description in the results section

Methods that may be used to describe a surveillance system 1.Review of documents 2.Review of products/ outputs 3.Qualitative interviews of key informants 4.Quantitative interviews of participants

1. Reviewing documents Consider: –Work plans –Guidelines –Operation manuals –Forms Generates information on the theoretical surveillance system

2. Reviewing products/ outputs Consider: –Records –Data –Feedback reports Generates information on the actual surveillance system

Elements of the surveillance system to describe (1/2) Diseases under surveillance Forms used Case definition Population under surveillance –Total, sentinel or special Reporting mode –Passive, stimulated or active

Elements of the surveillance system to describe (2/2) Data structure –Individual or aggregated Data transmission –Flow chart Data analysis procedures –Type, frequency Feedback –Type, frequency Action taken

Define objectives of a surveillance system Identify or help formulating the actual or desirable objective(s) of the surveillance system –Identify the goal of the programme served –Identify what decisions may need to be taken Review the attributes of the system Review whether the systems attributes are satisfactory to reach the objective(s)

Description of the needs –Country context –Disease patterns –Target audience Description of the surveillance system –Objectives –Process –Structure: resources, system Analysis of attributes Analysis of capacities /needs Implementation of prioritized modifications Plan of evaluation

Simplicity Flexibility Acceptability Data quality Sensitivity Positive predictive value Representativeness Timeliness Cost System attributes

Simplicity Structure –information needed –number and type of sources –number of information users Ease of operation –data transmission –system maintenance –data analysis –information dissemination

Flowchart for surveillance of HIV/AIDS in Norway

Ability of the system to accommodate changes with little additional time, persons or allocated funds –New event to follow-up –New case definition –New data about an event –New sources of information Flexibility

Acceptability Knowledge: –Case definition, notification procedures Notification conditions –Sufficient stock of notification forms –Working charge –Simplicity Motivation –Understanding the importance –Risk perception –Existence of a regular feedback

Data quality Completeness and validity of the data recorded Completenness of records –blanks in the surveillance forms Errors when computing data Influenced by –Simplicity of surveillance form –Clarity of electronic surveillance forms –Training –Validation

Sensitivity Two aspects: –proportion of cases of a disease or health condition detected by the surveillance system. –ability to detect outbreaks, to monitor changes (out of outbreaks, how many were detected) Requires (in principle) –to validate the information collected and –to collect information outside of the system to determine the frequency of the condition in the community.

Factors influencing case detection The probability of medical consultation –Severity of symptoms –Access to health care –Social and cultural values and habits The probability of clinical diagnosis –Pathogen causes a clinical condition also caused by other pathogens The probability of lab diagnosis –Demand for lab-testing –Existence of laboratories with the specific technique

Quantitative estimates of the sensitivity of a surveillance system Health care facility survey –Proportion of cases seen in the health care facility that are reported / captured by the surveillance system Population survey –Proportion of cases occurring in the population: Seen in health care facilities Reported / captured by the surveillance system Mathematical model –Cases captured /expected

Sensitivity of measles surveillance in Haora district, West Bengal, India Methods used –Cluster sample with door to door case search –Review of health care facility and surveillance records Key results: –Of 240 cases identified in the population, 8 (3.3%) were seen in public health care facilities and reported Conclusion: –Measles surveillance is only 3.3% sensitive Recommendation: –Promote vitamin A management in the population to increase the use of health system in the case of measles

Positive Predictive Value PPV depends on –sensitivity –specificity, and the –prevalence of the condition in the population Consequence of a low PPV –frequent "false-positive" report –inappropriate follow-up of non-cases –incorrect identification of epidemics (artifacts) –wastage of resources –inappropriate public concern (credibility)

Measles case definitions Suspect case –rash and fever Probable case –rash, fever, and (cough, coryza or conjunctivitis) –epidemiological link to a confirmed case Laboratory confirmed –saliva/serum IgM positive

Predictive value of notified measles Effect of change in incidence

Representative = accurately describing the distribution of a event health in the population by place and time Related to –data quality –bias of data collection –completeness of reporting Representativeness

Time Timeliness Delay / speed between steps in surveillance –onset –diagnosis –report –data entry –analysis –interpretation –intervention Chronic vs acute disease

Notification completed Notification received Intervention # days since onset of symptoms (n = 255) (n = 264) (n = 216) Median delay between notification and intervention by public health services Shigellosis, Delaware, 1991

Cost Cost is the estimated economic cost of the system Critical evaluation: –Could the system do an identical or better job while using less resources?

Sensitivity Representativeness Predictive value positive Timeliness Acceptability Flexibility Simplicity Cost Buehler's balance of systems attributes

Description of the needs –Country context –Disease patterns –Target audience Description of the surveillance system –Objectives –Process –Structure: resources, system Analysis of attributes Analysis of capacities /needs Implementation of prioritized modifications Plan of evaluation

Gaps between needs and capacities Is the system able to –adapt to changing needs –detect cases and outbreaks –investigate and intervene –disseminate and exchange information identification of weaknesses

Description of the needs –Country context –Disease patterns –Target audience Description of the surveillance system –Objectives –Process –Structure: resources, system Analysis of attributes Analysis of capacities /needs Implementation of prioritized modifications Plan of evaluation

Prioritisation of weaknesses to be addressed

Example 1 Surveillance of HIV/AIDS in Spain –notification on AIDS cases –aggregate reporting by laboratories on new HIV diagnoses –sentinel clinics report on HIV testing Which attributes are relevant?

Example 2 Surveillance for surgical site infections in a hospital –each patient after surgery is observed –complications are recorded on the observation card Which attributes are relevant?

Example 3 Surveillance collecting information on viral meningitis in Poland –detection of enteroviral outbreaks –investigation for poliovirus –monitoring tick-borne encephalitis foci –identification of new neurotropic viruses Which atributes are relevant?