1. TM Building Program Capacity for Evaluating Tuberculosis Surveillance Data Quality Presenter: Lakshmy Menon MPH ASPH/CDC Fellow Co-authors: Kai Young MPH, CHES Valerie Robison DDS, PhD Lori Armstrong PhD Division of Tuberculosis Elimination National Center HIV/AIDS, Viral Hepatitis, STD and Tuberculosis Prevention

2. TM Overview Division of Tuberculosis (TB) Elimination Mission Evaluation Background Gaps in Knowledge Framework: Quality Surveillance Data Evaluation Goals and Objectives Evaluation Stakeholders Evaluation Design

3. TM Division of Tuberculosis (TB) Elimination Mission Support a nationwide framework for surveillance of TB and evaluation of TB program performance Funds 68 state and territorial TB control programs Provide leadership and formulates national policies and guidelines

4. TM The Gaps in Knowledge What are the major characteristics of state and local TB programs’ surveillance systems? What surveillance activities take place? How does the data flow from one reporting level to the next? Are the surveillance systems strong and functional? Are data collected and reported via a program’s surveillance system accurate, complete, and timely?

5. TM Quality Surveillance Data Level 3 National TB Surveillance System Level 2 State TB Program Level 1 City/County TB Program Framework: Quality Surveillance Data Data Flow Between Reporting Levels ReliabilityPrecision IntegrityConfidentiality Timeliness Accuracy Completeness

6. TM Evaluation Goal Evaluate the quality of national tuberculosis surveillance data Engage stakeholders in quality assurance practices Build capacity at local/state level to conduct surveillance evaluations

7. TM 1.Evaluate the strength of a program’s surveillance system 2.Measure the accuracy, completeness, and timeliness of data reported to the National TB Surveillance System (NTSS) Evaluation Objectives

8. TM Engaging Stakeholders DTBE Program Consultants Program Evaluation Team Surveillance Team State TB Program Local Health Jurisdictions (LHJs)

9. TM Evaluation Team CDC Evaluation fellow Epidemiologist State Evaluation focal point Surveillance manager/epidemiologist Local Health Jurisdiction Surveillance manager/epidemiologist

10. TM Building Capacity State TB Program/LHJs Document data flow Identify + describe key surveillance activities Quantify surveillance process elements Identify implementation gaps Identify solutions to address gaps

11. TM Evaluation Design: Sampling Strategy Objective 1: Evaluate the strength of a surveillance system Convenience sampling to select participating programs States: low-, medium-, and high-burden 1 Big City Program

12. TM Evaluation Design: Sampling Strategy Objective 2: Measure the accuracy, completeness, and timeliness of data reported to the National TB Surveillance System (NTSS) Simple random sampling Population: Cases closed in 2006 or 2007 Random number table used to select sample cases using state case number Disease Burden Low  (5–10) Medium  (10–20) High  (20–30) State case numbers provided to state programs for inviting local jurisdictions

13. TM Objective 1: Evaluate the strength of a program’s surveillance system Desk Review On-site Review Activities

14. TM Functional Areas of a Strong Surveillance System I. Surveillance Infrastructure II. Standard Operating Procedures for Data Reporting III. Data Collection and Reporting Tools IV. Data Management Processes V. Data Dissemination Objective 1: Evaluate the strength of a program’s surveillance system 1. Structure 2. Function 3. Capacity Elements* of Each Functional Area Data Captured by Objective 1 Tools

15. TM Objective 1 - Scoring Template Surveillance System Functional AreaScore Calculation – From Desk and On-site Tools MeasureScore I. Surveillance InfrastructureColumn AColumn CTotal There is an organizational document that clearly identifies surveillance positions (data collection, data reporting, data entry, and data management) at all reporting levels Structure2/2 4/4 All positions related to surveillance activities are filled Function0/2 0/4 A training plan exists for data collection and reporting staff Function2/2 4/4

16. TM Evaluation Dashboard 1 Surveillance System Functional Areas – Indicators IIIIIIIVV Structure4/4Structure/2Structure/4Structure/14Structure/4 Function0/4Function/8Function/4Function/4Function/2 Capacity4/4Capacity /2Capacity Total: 8Total: Possible: 12Possible: 10Possible: 8Possible: 20Possible: 6

17. TM Objective 2: Measure the accuracy, completeness, and timeliness of data reported to the National TB Surveillance System (NTSS) Surveillance Data Quality Audit Abstract Data from Source Documents Calculate Accuracy, Completeness, and Timeliness Scores Activities

18. TM Evaluation Dashboard 2 Objective 2: Surveillance Data Quality Audit RVCT Data ItemAccuracy (2/1/0) Completeness (1/0) Timeliness (1/0) DOB Race Country of Origin Status at diagnosis of TB Previous Diagnosis of TB Initial DST Final DST Total/46/23/4

19. TM Funding received in Jul 2009 Inviting participants Piloting tools Dec 2009 –Jan 2010 Where are we now?

20. TM Acknowledgements Awal Khan PhD Kashef Ijaz MD Tom Navin MD Heather Duncan MPH Bob Pratt Carla Jeffries MPH Thanks!

21. TM Contact Information Presenter: Lakshmy Menon Email: LMenon@cdc.gov Telephone: 404.639.1694 Address: 1600 Clifton Rd NE MS E10 Atlanta, GA, 30329-1902

22. TM

23. Background Data accuracy audits in 3 city TB programs completed in 2005 Concordance of audited data varied greatly Very little surveillance process information available

24. TM Objective 1: Calculating Scores for Measures Functional Areas I – V are scored based on structure, function, and capacity (measures) Each measure receives a score ‘1’ if documentation is required and is available; otherwise ‘0’ ‘1’ if the element is implemented as planned; otherwise ‘0’ Scores are totaled in Excel workbook Indicators are reported by measure on Dashboard 1

25. TM Objective 2: Calculating Scores for Measures Accuracy ‘2’ if concordant ‘1’ if discordant Incomplete source documents ‘0’ Completeness ‘1’ if NTSS complete ‘0’ if incomplete Timeliness ‘1’ if reported within appropriate time period ‘0’ if not reported within appropriate time period Scores are calculated in Excel workbook Indicators are reported by element in Dashboard 2

26. TM City/County TB Program InputActivities Output Who: What: When: What: When: What: Input for State TB program Input to NTSS Activities Output Who: What: When: What: When: Sample Data Flow Diagram

27. TM Evaluation Tools Document request Desk review instrument On-site review survey instrument Data flow diagram Scoring template Dashboard 1 Data abstraction form Dashboard 2