1. Data Sources, Quality, Management, and HMIS Lisa V. Adams, MD E&E Regional Workshop Kiev, Ukraine May 23-26, 2006

2. E&E Regional Workshop for Monitoring and Evaluation of Tuberculosis Programs2 TB Data-Collection Methods and Sources  Routinely collected data (including global TB reporting)  Process monitoring and evaluation  Program evaluation/reviews  Special surveys

3. E&E Regional Workshop for Monitoring and Evaluation of Tuberculosis Programs3 Routine Recording (Facility Level)  TB Register  TB Treatment Card  Laboratory Register  Cough Register  Stock Cards and Consumption Reports

4. E&E Regional Workshop for Monitoring and Evaluation of Tuberculosis Programs4 Routine Reporting (District and National level)  District TB Register  Quarterly report of new cases and relapses of TB  Quarterly report on results of treatment of pulmonary TB patients registered 12-15 months earlier

5. E&E Regional Workshop for Monitoring and Evaluation of Tuberculosis Programs5 Process Monitoring and Evaluation  Analysis of recording and reporting  Triangulation (data verification)  Supervision  Observation  Records of trainings held, meetings held, events, job evaluations etc…

6. E&E Regional Workshop for Monitoring and Evaluation of Tuberculosis Programs6 Program Review  Comprehensive review of the entire program  Conducted every 1 to 5 years  External and internal experts break up into groups and cover a representative sample of the country  Usually provides input for developing or revising the medium term development plan

7. E&E Regional Workshop for Monitoring and Evaluation of Tuberculosis Programs7 Special Studies  Prevalence survey  Population-based survey  Facility surveys  Vital-registration surveys  Tuberculin surveys  Drug-resistance surveys  Action research

8. E&E Regional Workshop for Monitoring and Evaluation of Tuberculosis Programs8 Qualitative vs. Quantitative Quantitative  Measure (how many, how much?)  Determine causal relationships  Representative for target population  Sampling: statistical significans  Technique: structured  Present: tables/graphs Qualitative  Describe (why, how, what in its context)  Identify relationships between factors and issues  Identify variation in a specific context  Sampling: inclusion of all relevant variations  Technique: in-depth, semi structured, participatory  Present: descriptive information

9. E&E Regional Workshop for Monitoring and Evaluation of Tuberculosis Programs9 Example of a National Level Data-Collection System 2000 200220042006 Routine information system & surveillance Facility survey Facility survey Prevalence Survey DRS External Monitoring Visits Facility survey review Special study

10. E&E Regional Workshop for Monitoring and Evaluation of Tuberculosis Programs10 Basis for Decision-Making

11. E&E Regional Workshop for Monitoring and Evaluation of Tuberculosis Programs11 Data Quality Why is data quality important?  To enhance evidence-based decision-making  To give substance and weight to recommendations for future actions Issues of Data Quality:  Reliability  Validity  Completeness  Relevance  Timeliness

12. E&E Regional Workshop for Monitoring and Evaluation of Tuberculosis Programs12 GI=GO: More possibilities: Quality In-Garbage Out (QIGO) Quality In-Little Out (QILO) Quality In-Quality Out (QIQO) Garbage management Garbage In = Garbage Out

13. E&E Regional Workshop for Monitoring and Evaluation of Tuberculosis Programs13 Data Quality Assessments At district level: during supervision:  Step 1: interview appropriate individuals to assess understanding of 1.the usefulness/value of HMIS 2.data collection, analysis, and maintenance process  Step 2: review reports to determine whether they are consistent

14. E&E Regional Workshop for Monitoring and Evaluation of Tuberculosis Programs14 Data-Quality Assessments (con’t)  Step 3: Periodically sample and review data for completeness, accuracy, and consistency 1.Indicators and definitions are consistent with NTP guidelines 2.Data collection is consistent from year to year 3.Data are complete in coverage 4.Formula used to calculate indicator (if any) is applied correctly

15. E&E Regional Workshop for Monitoring and Evaluation of Tuberculosis Programs15 Data-Quality Assessments (con’t)  Step 4: Compare and triangulate various sources (e.g., central-office records with district or district with facility for consistency and accuracy)  Combine qualitative and quantitative data Source 1 Source 3Source 2 Truth

16. E&E Regional Workshop for Monitoring and Evaluation of Tuberculosis Programs16 From Data to Information: Basic Epidemiology  Collect relevant data for defined indicators  Ask key questions  From absolute to relative data (%, rates)  Analyze  Use solid epidemiological reasoning and practice  Compare indicators in time, place, person, and with target  Identify gaps, high-risk populations

17. E&E Regional Workshop for Monitoring and Evaluation of Tuberculosis Programs17 From Information to Evidence  Accurate interpretation of data (relevance, coherence, context)

18. E&E Regional Workshop for Monitoring and Evaluation of Tuberculosis Programs18 From Evidence to Decision- Making: Using the Information  Getting on the political agenda of the right decision-maker at the right time  Take user perspective into account (politician, technician, public)  Use effective presentation  Apply advocacy skills

19. E&E Regional Workshop for Monitoring and Evaluation of Tuberculosis Programs19 The Issue is not that we need more data….... but better use of data

20. E&E Regional Workshop for Monitoring and Evaluation of Tuberculosis Programs20 HMIS: Definition  An HMIS is a system of record-keeping, reporting, processing, analysis, interpretation, use, and feedback of information.

21. E&E Regional Workshop for Monitoring and Evaluation of Tuberculosis Programs21 HMIS: Functions -Provides different levels of beneficiaries (clients, community, service-providers, managers, planners and policy makers) with timely and relevant information -Used to formulate policy, to plan, implement, monitor, supervise, and evaluate health- related activities

22. E&E Regional Workshop for Monitoring and Evaluation of Tuberculosis Programs22 HMIS Provides Information On:  Health status (through surveillance and surveys)  Health-related interventions and services (activities, progress, quality) through surveillance, program monitoring, and evaluation

23. E&E Regional Workshop for Monitoring and Evaluation of Tuberculosis Programs23 Purposes of HMIS: Evidence-Based Decision Making -Monitoring and evaluation of control programs -Plan actions, programs and resources -To prioritize the allocation of health resources -To provide the basis for epidemiological research -Accountability

24. E&E Regional Workshop for Monitoring and Evaluation of Tuberculosis Programs24 Management Cycle Where are we now? Situation Analysis Where do we want to go? Objectives Where did we reach? Evaluation Which route shall we take? Strategy Is everything going according to plan? Monitoring Identifying constraints Taking action How shall we travel there? Plan of Activities HMIS

25. E&E Regional Workshop for Monitoring and Evaluation of Tuberculosis Programs25 Problems in HMIS 1: Design 1. Top-down control, leaving out lower levels 2. Unclear objectives and targets 3. Too much working time spent on HMIS 4. No guidelines 5. Not enough skills 6. Not enough staff 7. Fragmentation in services

26. E&E Regional Workshop for Monitoring and Evaluation of Tuberculosis Programs26

27. E&E Regional Workshop for Monitoring and Evaluation of Tuberculosis Programs27 Problems in HMIS:2: Collection  Too many records  Too late  Poor quality  Poor layout, confusing terminology  Incomplete  Exclusion of private and hospitalized patients?  Inflexible data collection  Resistance to change

28. E&E Regional Workshop for Monitoring and Evaluation of Tuberculosis Programs28 Problems in HMIS 3: (Self-) Assessment  Data not analyzed to produce information, but just forwarded upward  No indicators to assess program progress at different levels  Presentation not relevant, out of date  Management culture does not value information (decision-making on intuitive or political basis only)  Improper political use

29. E&E Regional Workshop for Monitoring and Evaluation of Tuberculosis Programs29 Problems in HMIS 4: Use  Inadequate training for managers  Data not available for the right persons at the right time  No feedback to lower levels, no pressure to use data  Data not used for supervision  Rigid/inflexible interpretation (success rate in the era of HIV)

30. E&E Regional Workshop for Monitoring and Evaluation of Tuberculosis Programs30 A Good HMIS System:  Provides all required information  Is easy to implement, well-organized, and efficient  Requires minimal staff time  Is consistent with other management and recording systems in place  Is useful for all stakeholders and used appropriately

31. E&E Regional Workshop for Monitoring and Evaluation of Tuberculosis Programs31 Group Work: HMIS Challenges Question 1a: What are the main challenges in your country regarding the design of your HMIS? Question 1b: What solutions do you propose? Question 2a: What are the main challenges in your country regarding the collection and quality of data that you receive? Question 2b: What solutions do you propose? Question 3a: What are the main challenges in your country regarding maximizing use of data collected for NTP management? Question 3b: What solutions do you propose?

32. E&E Regional Workshop for Monitoring and Evaluation of Tuberculosis Programs32 Group work: HMIS challenges Vopros 1a: Kakiye glavnyje problemy v vashey strane v otnosheniyi postroyeniya vashey SIUZ? Vopros 1b: Kakiye resheniya vy predlagayete? Vopros 2a: Kakiye osnovnyje problemy v vashey strane v otnosheniyi sbora i kachestva poluchayemykh vami dannykh? Vopros 2b: Kakiye resheniya vy predlagayete? Vopros 3a: Kakiye osnovnyje problemy v vashey strane v otnosheniyi maksimizatsiyi ispol’zovaniya dannykh, sobrannykh dlya upravleniya NTP? Vopros 3b Kakiye resheniya vy predlagayete?

33. E&E Regional Workshop for Monitoring and Evaluation of Tuberculosis Programs33 Facility Visit  Understand the data-collection process of the TB facility  Collect specific information on TB from the clinics using your worksheets  Calculate and interpret the data you’ve collected (using the TB-compendium) if necessary  Report back to the group