Indicators, Data Sources, and Data Quality for TB M&E

Slides:



Advertisements
Similar presentations
SURVEY QUALITY CONTROL
Advertisements

MICS4 Survey Design Workshop Multiple Indicator Cluster Surveys Survey Design Workshop Survey Quality Control.
Introduction to Monitoring and Evaluation
Data Quality Considerations
Donald T. Simeon Caribbean Health Research Council
Quality Data for a Healthy Nation by Mary H. Stanfill, RHIA, CCS, CCS-P.
Preparing Data for Quantitative Analysis
Data Processing Topic 4 Health Management Information Systems João Carlos de Timóteo Mavimbe Oslo, April 2007.
Collecting High-Quality Data. M&E Plan FrameworkIndicators Data Collection Data Quality Data Use and Reporting Evaluation Strategy Budget Part of the.
Monitoring and Evaluation: Tuberculosis Control Programs
Compendium of Indicators for Monitoring and Evaluating National Tuberculosis Programs Using the Compendium to Plan for Monitoring and Evaluation of NTPs.
Comprehensive M&E Systems
Compendium of Indicators for Monitoring and Evaluating National Tuberculosis Programs.
Chapter Three Research Design.
1 Module PLANNING AND ORGANIZING EXTERNAL QUALITY ASSESSMENT.
Unit 5: Core Elements of HIV/AIDS Surveillance
Short Course on Introduction to Meteorological Instrumentation and Observations Techniques QA and QC Procedures Short Course on Introduction to Meteorological.
Unit 4: Monitoring Data Quality For HIV Case Surveillance Systems #6-0-1.
FINAL REPORT: OUTLINE & OVERVIEW OF SURVEY ERRORS
Dimensions of Data Quality M&E Capacity Strengthening Workshop, Addis Ababa 4 to 8 June 2012 Arif Rashid, TOPS.
Module 3: Final Case Study # 3-CS-1. Case Study: Instructions v Try this case study individually. v We’ll discuss the answers in class. # 3-CS-2.
CPA is a UKAS company The Assessment Process 2014 Seminars.
Evaluation Methods.
1 Interpretation and use. 2 The walls inside are plastered with laboriously made graphs…
RELIABILITY AND VALIDITY OF DATA COLLECTION. RELIABILITY OF MEASUREMENT Measurement is reliable when it yields the same values across repeated measures.
Sampling : Error and bias. Sampling definitions  Sampling universe  Sampling frame  Sampling unit  Basic sampling unit or elementary unit  Sampling.
RESEARCH A systematic quest for undiscovered truth A way of thinking
C M Clarke-Hill1 Collecting Quantitative Data Samples Surveys Pitfalls etc... Research Methods.
Sina Keshavaarz M.D Public Health &Preventive Medicine Measuring level of performance & sustaining improvement.
1 What are Monitoring and Evaluation? How do we think about M&E in the context of the LAM Project?
1 Module OVERVIEW OF EXTERNAL QUALITY ASSESSMENT.
Unit 10. Monitoring and evaluation
Ensuring the Fundamentals of Care in Family Planning and Reproductive Health Services MODULE 2 Facilitative Supervision for Quality Improvement Curriculum.
Ensuring the quality of Qualitative Data Presented by Lorie Broomhall, Ph.D. Senior HIV/AIDS Advisor Nigeria Monitoring and Evaluation Management Services.
DATA QUALITY How closely do the data used reflect the truth about results?
Unit 3: Universal Case Reporting and Sentinel Surveillance for STIs
School of Health Systems and Public Health Monitoring & Evaluation of HIV and AIDS Programs Data Quality Wednesday March 2, 2011 Win Brown USAID/South.
Data Quality Assessment
1 ON-SITE SUPERVISION – CHECKLISTS. 2 Checklists Serve as documentation of the visit and record of current conditions and actions needed Help carrying.
1 The Good, the Bad, and the Ugly: Collecting and Reporting Quality Performance Data.
Data Processing & Data Quality. A virtuous cycle The implicit assumptions underlying information systems are twofold: first, that good data, once available,
Consultant Advance Research Team. Outline UNDERSTANDING M&E DATA NEEDS PEOPLE, PARTNERSHIP AND PLANNING 1.Organizational structures with HIV M&E functions.
Data Quality Assessment of PEPFAR ART Sites In Nigeria Final Report February 17, 2006 Nigeria/Monitoring and Evaluation Management Services in collaboration.
INTRODUCTION TO INFORMATION SYSTEMS FOR IMMUNIZATION SERVICES IPV Global Workshop March 2014.
Comprehensive M&E Systems: Identifying Resources to Support M&E Plans for National TB Programs Lisa V. Adams, MD E&E Regional Workshop Kiev, Ukraine May.
First meeting of the Technical Cooperation Group for the Population and Housing Censuses in South East Europe Vienna, March 2010 POST-ENUMERATION.
Data Preparation and Description Lecture 24 th. Recap If you intend to undertake quantitative analysis consider the following: type of data (scale of.
United Nations Expert Group Meeting on International Standards for Civil Registration and Vital Statistics Systems, June 2011, New York Evaluation.
Session 6: Data Flow, Data Management, and Data Quality.
Session 2: Developing a Comprehensive M&E Work Plan.
Assessing Logistics System Supply Chain Management 1.
A Training Course for the Analysis and Reporting of Data from Education Management Information Systems (EMIS)
Data Quality Management. Learning Objectives  At the end of this session, participants will be able to:  Summarize basic terminology regarding data.
Compendium of Indicators for Monitoring and Evaluating National Tuberculosis Programs.
CRITICALLY APPRAISING EVIDENCE Lisa Broughton, PhD, RN, CCRN.
MEASURE Evaluation Data Quality Assurance Workshop Session 3 Introduction to Routine Data Quality Assessment.
USAID’s Data Quality Standards & Conducting a DQA # ? % ?
Data Sources, Quality, Management, and HMIS Lisa V. Adams, MD E&E Regional Workshop Kiev, Ukraine May 23-26, 2006.
11 ii. Develop a plan for aDSM
Unit 4: Monitoring Data Quality For HIV Case Surveillance Systems
Dissemination Workshop for African countries on the Implementation of International Recommendations for Distributive Trade Statistics May 2008,
Chapter Three Research Design.
IX- PREPARING THE BUDGET
Monitoring and Evaluation
Measuring Data Quality
monitoring & evaluation THD Unit, Stop TB department WHO Geneva
11 iii. Define management and supervision roles and responsibilities
Indicators, Data Sources and Data Quality for TB M&E
Internal Control Internal control is the process designed and affected by owners, management, and other personnel. It is implemented to address business.
Monitoring and Evaluation in Communication Management
Presentation transcript:

Indicators, Data Sources, and Data Quality for TB M&E This presentation discusses some of the most important factors influencing which indicators we use. Important to review this topic before doing the indicator selection activity. Data sources differ in every context, country, and region so as I go through this presentation, please try to think of data sources and data quality issues that your country uses and has trouble with, so that we can discuss them after the presentation.

Criteria of good indicators Valid: indicator actually measures the phenomenon it is intended to measure Reliable: indicator produces the same results when used repeatedly, so it always measures the same phenomenon Specific: indicator measures only the phenomenon it is intended to measure Sensitive: indicator reflects changes in the phenomenon under study Operational: indicator is measured with developed and tested definitions and reference standards These ideas are probably review and were discussed in “how to use the compendium.” Indicators are signs, clues, and markers as to how close we are to our path and how things are changing.

Qualitative vs. Quantitative Qualitative: indicators answer questions about how well the program elements are being carried out Quantitative: indicators measure how much and how many Examples of qualitative M&E tools: Sign-in logs, Registrations, forms, registers, checklists Program activity forms Patient charts Structured questionnaires Qualitative M&E tools: Focus group Direct observation

Factors in Indicator Selection What national, district and local levels need to know Availability of the data Availability of human and financial resources to manage the data Program needs Lender requirements

TB data collection methods and sources Routinely collected data Process monitoring and evaluation Program evaluation/reviews Global TB reporting Special surveys Once you have designed a framework and picked appropriate indicators, data collection strategy needs to be developed. Of course, you should have been thinking of data availability while developing the framework and indicators but now you can focus on collection and analysis No single source can satisfy data needs for M&E.

Routine Recording TB Register TB Treatment Card Laboratory Register Cough register These are the basic units of data collected at TB treatment facilities and microscopy units. To ensure data quality, it is important that: These cards are filled out correctly These units are filled out in the same way What this often requires is: An easy-to-use card (pre-tested) Training Frequent review of how nurses, lab techs are filling them out (supervision and feedback) Really communicating why it is important to have this information filled out correctly and timely

Routine Reporting District TB Register Quarterly report of new cases and relapses of TB Quarterly report on results of treatment of pulmonary TB patients registered 12-15 months earlier Monthly or quarterly reporting forms are sent to Basic Management Unit (BMU), where they are aggregated and then sent to the higher level. Assuming the basic units are filed out correctly, we want to ensure that the “transfer” to the aggregated level is done correctly and in the same fashion. Another issue is timeliness. Again, requires supervision and feedback.

Process Monitoring and Evaluation Analysis of recording and reporting Supervision Records of trainings held, meetings held, events, etc…

Program Evaluation/Review Comprehensive review of the entire program Conducted every 2-5 years External and internal experts break up into groups and cover a representative sample of the country Usually provides input for developing or revising the medium-term development plan

Global Reporting: estimated incidence, case notification, treatment outcomes, some budget information and coverage. Can use the numbers from WHO to track your own country over time. Source: WHO (2004). Global Tuberculosis Control: Surveillance, Planning, Financing.

Special studies Prevalence survey Population-based survey Facility surveys Vital-registration surveys Tuberculin surveys Drug-resistance surveys Read from blurbs on special studies in the intro part of the compendium.

Example of a national level data-collection system Prevalence Survey Prevalence Survey DRS DRS Facility survey Facility survey Facility Survey DRS – Drug Resistance Survey If appropriate, all of these activities should be budgeted for in a GFATM proposal A facility survey could be considered a TB-Program Review External Monitoring Visits Routine information system & surveillance 2000 2002 2004 2006

Why is data quality important? The primary function of health-information systems is to provide data that enhance decision-making in the provision of health services. By ensuring high-quality data, the health information system attempts to guarantee that decision-makers have access to unbiased and complete information Your data source is only as good as the data it produces. While ensuring that the importance of data quality has been addressed in the previous slides, it is important to spend a few moments thinking about this issue of good data specifically. It isn’t just for feeding information into an indicator that needs to be reported elsewhere, it is important to see these as tools for decision makers at national, district, and local levels to improve their programs.

Standards for good quality data Validity: Do the data clearly and directly measure what was intended to measure? Integrity: Are mechanisms in place to reduce the chance that data are intentionally manipulated? Precision: Are the data at the appropriate level of detail? Reliability: Would you come to the same finding if the data collection and analysis were repeated? Timeliness: Are data available frequently enough to inform decisions?

Impediments to good data quality Inappropriate data-collection instruments and procedures Poor reporting and recording Errors in processing data (editing, coding, data entry, tabulating)

What can be done to improve and ensure data quality? Keep the design of the information system as simple as possible Involve users in the design of the system Standardize procedures and definitions Pre-test data-collection instruments to make sure they are useful and user-friendly Ensure that data collected are useful to the data collector Regular supervision and feedback from supervisors Plan for effective checking procedures (such as cross-checking) Training (data collection instruments, data processing, analysis and decision-making based on evidence)

Data-quality assessments Example at district level: Step 1: Interview appropriate individual to obtain understanding of data collection, analysis, and maintenance process Step 2: Review reports to determine whether they are consistent Now that we have discussed basic kinds of data sources and the importance of ensuring data quality, let’s walk through addressing data quality in a step-wise fashion.

Data-quality assessments (con’t) Step 3: Periodically sample and review data for completeness, accuracy, and consistency Indicator definitions are consistent with NTP guidelines Data collection is consistent from year to year Data are complete in coverage Formula used to calculate indicator (if any) is applied correctly Step 4: Compare central-office records with district or district with facility for consistency and accuracy

Data-quality assessments (con’t) Possible data-quality limitations: Validity: The reported data do not accurately represent the population. For example, records may over-report or underreport certain parts of the population Integrity: The data could be manipulated for a variety of reasons Timeliness: If reporting is not up to date, than decisions may not based on most recent evidence Reliability: Implementation of data collection may be irregular or mistimed

Conclusion needed based on context/region and what the biggest issue is