Unit 10. Monitoring and evaluation Photo from: WHO. Revised TB recording and reporting forms, Version 2006. WHO/HTM/TB/2006.373 http://www.who.int/tb/dots/r_and_r_forms/en/index.html TB Infection Control Training for Managers at the National and Subnational Levels

Objectives After this unit, the participant will be able to: Define monitoring and how it is essential to supervision List two indicators to use in monitoring infection control at the national level, and two for use at the facility level [Review slide]

Outline Definitions: monitoring vs. evaluation Types of indicators: input, output, outcome and impact Choice of indicators Examples of indicators at global, national and facility levels Setting targets [Review slide]

Develop the plan (to include goals and objectives) Lifecycle of IC plan Develop the plan (to include goals and objectives) Implement This is the same diagram we discussed in the Managerial unit. Like other good plans, the IC plan has goals and objectives. As a plan is being implemented, it will need to be fine-tuned as the situation changes. Monitoring helps the manager decide what revisions are needed in the implementation, so objectives are met. Monitor, evaluate Revise

Monitoring is…. The routine tracking and reporting of priority information about a programme or project, its inputs and intended outputs, outcomes, and impact [Review slide] We’ll define inputs, outputs, outcomes and impact in a few minutes. Citations: WHO. A guide to monitoring and evaluation for collaborative TB/HIV activities. 2009. WHO/HTM/TB/2009.414, WHO/HTM/HIV/09.01 http://www.who.int/tb/publications/2009/en/index.html Global Fund to Fights AIDS, TB and Malaria. Monitoring and evaluation toolkit-2009 version. http://www.theglobalfund.org/en/me/guidelines_tools/

Monitoring Uses data obtained through record keeping, regular reporting, surveillance systems, observation or surveys Helps managers determine which areas require greater effort Is essential for supervision Is also used to measure trends over time, so methods need to be consistent to ensure appropriate comparison [Review slide] Citations: WHO. A guide to monitoring and evaluation for collaborative TB/HIV activities. 2009. WHO/HTM/TB/2009.414, WHO/HTM/HIV/09.01 http://www.who.int/tb/publications/2009/en/index.html Global Fund to Fights AIDS, TB and Malaria. Monitoring and evaluation toolkit, 2009 version. http://www.theglobalfund.org/en/me/guidelines_tools/

Evaluation is… The rigorous, scientifically-based collection of information about the program activities, characteristics, and outcomes that determine the merit or worth of the programme [Review slide] Citation: WHO. A guide to monitoring and evaluation for collaborative TB/HIV activities. 2009. WHO/HTM/TB/2009.414, WHO/HTM/HIV/09.01 http://www.who.int/tb/publications/2009/en/index.html Global Fund to Fights AIDS, TB and Malaria. Monitoring and evaluation toolkit, 2009 version. http://www.theglobalfund.org/en/me/guidelines_tools/

Evaluation studies Provide credible information for use in improving programmes, identifying lessons learned, and information decisions about future resource allocation [Review slide] [Ask participants:] Can you name some ways that monitoring and evaluation are different? [Possible responses]: The main purpose of monitoring is to help managers routinely monitor progress and make needed corrections during programme implementation. By contrast, evaluations are done to determine the worth of a program. While monitoring is regular and ongoing, Evaluation studies are done less frequently. They also use different methods. Monitoring uses routinely collected information. Evaluation requires a special study (although it should leverage any data or surveys are are regularly undertaken). Citations: WHO. A guide to monitoring and evaluation for collaborative TB/HIV activities. 2009. WHO/HTM/TB/2009.414, WHO/HTM/HIV/09.01 http://www.who.int/tb/publications/2009/en/index.html Global Fund to Fights AIDS, TB and Malaria. Monitoring and evaluation toolkit, 2009 version. http://www.theglobalfund.org/en/me/guidelines_tools/

M&E framework long term, cumulative effect of the programme over time Impact long term, cumulative effect of the programme over time Outcome short and medium term effects of the outputs Output products or deliverables of the programme’s interventions We’ll work our way from the bottom to the top of this framework. For a programme to achieve its goals and objectives, inputs (financial, human and material resources) must result in outputs (products or deliverables) If the outputs are achieved, the programme will have a positive outcome, which is a short and medium term effect. These outcomes should have a long term impact. In this monitoring and evaluation (M&E) framework, the blue boxes are linked (yellow arrows) in a chain of results. Citations: WHO. A guide to monitoring and evaluation for collaborative TB/HIV activities. 2009. WHO/HTM/TB/2009.414, WHO/HTM/HIV/09.01 http://www.who.int/tb/publications/2009/en/index.html Global Fund to Fights AIDS, TB and Malaria. Monitoring and evaluation toolkit, 2009 version. http://www.theglobalfund.org/en/me/guidelines_tools/ Diagram adapted from: Dominique Mathiot Input financial, human and material resources

Example of a results chain Impact Fewer health care workers with TB Outcome Coughing patients are rapidly identified, separated, and served Output HIV clinics have written triage procedures for coughing patients Our example is a country which has identified HIV clinics as high risk settings for TB transmission. Let’s work our way from top to bottom of the diagram; we’ll follow the chain of results first using the white boxes. One of the overall goals of the country’s IC plan is to decrease the spread of TB in order to protect health care workers and other patients. To have this impact, the country wants to be sure that coughing patients are rapidly identified, separated, and served. This will require written procedures be in place for coughing patients, which in turn requires there be an IC focal person identified and trained in each facility. We can check our logic by working our way back up the results chain by following the yellow arrows from bottom to top. We want to be sure that inputs, lead to outputs, which lead to outcomes and finally the desired impact. Starting at the bottom are the resources needed. One input is human resources: IC focal point staff identified in each HIV clinic. They participate in an IC training, including a workshop to finalize a national template for a clinic triage policy. The output is written triage procedures in each clinic. The trained IC focal point staff take the national template, adapt it to their settings, and put in place a written triage policy specific to their clinic. The short term outcome should be that coughing patients are identified, separated, and rapidly served. Over the long run, the impact will be less TB spread in the clinics. Fewer health care workers will have TB. Citations: WHO. A guide to monitoring and evaluation for collaborative TB/HIV activities. 2009. WHO/HTM/TB/2009.414, WHO/HTM/HIV/09.01 http://www.who.int/tb/publications/2009/en/index.html Global Fund to Fights AIDS, TB and Malaria. Monitoring and evaluation toolkit, 2009 version. http://www.theglobalfund.org/en/me/guidelines_tools/ Diagram adapted from: Dominique Mathiot Input Infection Control focal point staff identified and trained

Indicators Provide useful tools for supervision at all levels Are we doing the right things? Are we doing them right? Help managers and supervisors determine: What’s going well and should be reinforced and replicated elsewhere? What needs improvement and how can the gaps be addressed? [Review slide] Citation Global Fund to Fights AIDS, TB and Malaria. Monitoring and evaluation toolkit, 2009 version. http://www.theglobalfund.org/en/me/guidelines_tools/

Choose indicators that: Are able to measure performance Provide strategic information needed to make good decisions for managing and improving programme performance Are reliable and measurable on a regular basis Are clear, with well defined numerators and denominators using standard definitions [Review slide] The next few slides present examples of indicators. You can chose those that are the most useful for supervision of the IC programme at each level. Citation Global Fund to Fights AIDS, TB and Malaria. Monitoring and evaluation toolkit, 2009 version. http://www.theglobalfund.org/en/me/guidelines_tools/ WHO. A guide to monitoring and evaluation for collaborative TB/HIV activities. 2009. WHO/HTM/TB/2009.414, WHO/HTM/HIV/09.01 http://www.who.int/tb/publications/2009/en/index.html

Standardized IC indicators for collaborative TB/HIV activities Proportion of health-care facilities providing services for people living with HIV that have infection control practices that include TB control Proportion of health-care workers, employed in facilities providing care for people living with HIV, who developed TB during the reporting period WHO recently developed standardized, internationally accepted indicators for monitoring TB/HIV collaborative activities. Here are the two that address IC. While specific to facilities providing care for people living with HIV, they can be applied to any facility with a risk of TB transmission. The first is the proportion of health facilities that have IC practices. The second is the proportion of health-care workers who developed TB. The citation in the Instructor’s notes [see below] provides definitions for these 2 indicators, suggests how to collect the information, and assesses the strengths and limits of these indicators. Citation WHO. A guide to monitoring and evaluation for collaborative TB/HIV activities. 2009. WHO/HTM/TB/2009.414 http://www.who.int/tb/publications/2009/en/index.html

Global IC indicators and targets By 2012, 50% of countries should have Developed a national TB infection control plan Set up national surveillance of TB disease among health workers Assessed major health-care facilities and congregate settings for TB IC Reported on the implementation of TB infection control By 2013, all countries As we discussed in the introduction to the class, here are global IC indicators in the 2009 WHO Policy on Infection Control. The targets are for 50% of countries to achieve them by 2012, and all countries by 2013. [Review slide] Citation WHO. WHO policy on TB infection control in health-care facilities, congregate settings, and households. 2009. WHO/HTM/TB/2009.419 http://www.who.int/tb/publications/2009/en/index.html

Examples of national IC indicators How many health care workers had TB in the past year? Have health care workers been trained in the past year? Of the country’s tertiary (referral) hospitals, how many have: A person in charge of TB infection control? A TB IC assessment done in the past year? Training on TB IC conducted in the past year? Here are examples of infection control indicators for the national level. These are part of WHO’s TB data collection form countries completed in 2009. [Review slide]

Examples of facility level IC indicators Facility level plan in place Person responsible for TB IC Health facility assessment done Training conducted Triage and separation of TB patients # TB cases or suspects missed at intake Adequate ventilation and/or UVGI Use of respirators by HCWs in high risk settings [Review slide] “Missed at intake” means that TB symptoms or history were detected only after the patient enters a private room with a clinician, or after numerous visits while symptomatic, instead of upon first entry into the facility. Citation WHO. TB infection-control in the era of expanding HIV care and treatment. Addendum to WHO Guidelines for the prevention of TB in health are facilities in resource-limited settings. 2006 http://www.who.int/tb/publications/2006/en/index.html

Facility-level indicators Impact Number of health care workers with TB Outcome Number of TB cases or suspects missed at intake to the facility Output Facility-specific, written triage procedure exists Let’s go back to our example. Here are some indicators that could be used to during a supervisory visit to a district clinic to monitor the implementation of this part of the country’s IC plan. [Review slide] [Ask participants] Which of these indicators may also be useful for a national manager to assess the country-wide implementation of this part of the national IC plan? [Possible responses] Number of facilities with IC focal persons Number of facilities with written triage procedures to identify and separate coughing patients Number of health care workers with TB Citations: WHO. A guide to monitoring and evaluation for collaborative TB/HIV activities. 2009. WHO/HTM/TB/2009.414, WHO/HTM/HIV/09.01 http://www.who.int/tb/publications/2009/en/index.html Global Fund to Fights AIDS, TB and Malaria. Monitoring and evaluation toolkit, 2009 version. http://www.theglobalfund.org/en/me/guidelines_tools/ Diagram adapted from: Dominique Mathiot Input Infection control focal person is identified and trained

Setting targets Gap analysis: baseline compared to need not yet met Constraints to meeting the need Feasibility for scaling up Setting ambitious yet realistic targets is an important element of the planning process. Good targets are based on a comprehensive and up to date analysis of the situation. A gap analysis compares the baseline to the need. Setting targets also requires consideration of constraints to meeting the need, and feasibility for scale up. Citation Global Fund to Fights AIDS, TB and Malaria. Monitoring and evaluation toolkit, 2009 version. http://www.theglobalfund.org/en/me/guidelines_tools/

Example of setting a target By 2015, __ of the country’s HIV clinics will have a written policy for identification and separation of coughing patients Gap analysis: baseline is 0, need is 100% Constraints: no template policy, limited training capacity, few IC focal persons in the HIV clinics Feasibility: resources available to address the constraints and fill the gap Let’s work through an example. You have chosen this outcome indicator (in bold), and need to set the target (to fill in the blank). When you conduct your gap analysis, you find that none of the HIV clinics have a policy, but all need them. You face significant constraints in getting there: there is no sample policy in your country the clinics can use, there are few IC focal persons identified in the HIV clinics, and you have limited capacity to train them on adopting the policy. To set the target, you think through how well you can address these constraints by 2015, and thus, how much of a scale up is feasible. You’ll judge for your own country what is the appropriate target. Citation Global Fund to Fights AIDS, TB and Malaria. Monitoring and evaluation toolkit, 2009 version. http://www.theglobalfund.org/en/me/guidelines_tools/

Summary Monitoring is the routine tracking and reporting of priority information about a programme Use global indicators to allow measurement of impact across countries Chose additional indicators that provide strategic information needed to make good decisions for managing and improving programme performance Adapt standardized IC indicators to your country context, and use them for supervision at each level [Review slide]