1. Monitoring and Evaluation: A Review of Terms
Lisa V. Adams, MD
E&E Regional Workshop
Kiev, Ukraine
May 23-26, 2006

2. Goals
To provide better treatment for people with tuberculosis in Country X
To achieve a treatment success rate of 85% in Country X by the end of 2006
The core of any M&E system is the goals and objectives of the program. If they are written in such a way that they can be easily distinguished from one another and measured, the job of the m&e specialist will be much easier.
Goal: statement that describes the hoped for result of the program. Goals are achieved over the long term and through the combined efforts of multiple programs.
Tuberculosis Technical Focus Area Activties and Update All-Staff Meeting 2006

3. Objectives
To increase the number of pharmacies with adequate supplies of drugs
To increase to 100% the proportion of pharmacies implementing NTP guidelines on drug management by the end of 2005
Specific and operationalized statements detailing the desired accomplishments of a program (Rossi & Freeman, 1993)
A properly stated objective is action oriented and starts with the word: to and is followed by an action verb. It addresses what and when, but not how or why.
Tuberculosis Technical Focus Area Activties and Update All-Staff Meeting 2006

4. Objectives – Be SMART Specific Measurable Achievable Realistic
Time-Bound
note to trainer: useful to go through examples of each one of these components
Specific – does it cover only one rather than multiple activities?
Measurable – can it be counted in some way?
Achievable– can we achieve this with available resources? Is it doable?
Realistic – we would all love to detect and cure every case of TB in the world.
Time-Bound– does it give a time frame?
Transition – now we will discuss specific program components and how they described in monitoring and evaluation terms.
Tuberculosis Technical Focus Area Activties and Update All-Staff Meeting 2006

5. What do you need to implement the program?
Inputs
What do you need to implement the program?
A collection of resources (financial, personnel, equipment) which are the raw materials for the program
Tuberculosis Technical Focus Area Activties and Update All-Staff Meeting 2006

6. What do you need to implement the program?
Inputs
What do you need to implement the program?
Drugs and supplies
Clinical sites
Laboratories
Nurses, doctors, epidemiologists, etc
DOTS strategy, policies supporting DOTS
Money
A collection of resources (financial, personnel, equipment) which are the raw materials for the program
Tuberculosis Technical Focus Area Activties and Update All-Staff Meeting 2006

7. WHAT DOES THE PROGRAM DO?
Process
WHAT DOES THE PROGRAM DO?
A set of activities in which program inputs are utilized in pursuit of the results expected from the program.
Notice, these can be concrete, tangible things or can be conceptual– supervision, for example. Usually action oriented, for example, training or supervision activities that will produce something.
Tuberculosis Technical Focus Area Activties and Update All-Staff Meeting 2006

8. WHAT DOES THE PROGRAM DO?
Process
WHAT DOES THE PROGRAM DO?
Diagnosis and Treatment
Meetings
Training
Workshops
Reporting & Recording
Supervision
A set of activities in which program inputs are utilized in pursuit of the results expected from the program.
Notice, these can be concrete, tangible things or can be conceptual– supervision, for example. Usually action oriented, for example, training or supervision activities that will produce something.
Tuberculosis Technical Focus Area Activties and Update All-Staff Meeting 2006

9. WHAT DID THE PROGRAM ACHIEVE?
OUTPUTS
WHAT DID THE PROGRAM ACHIEVE?
The results obtained at the program level through the execution of activities using program resources.
When we think of outputs, we ask – what did the program achieve? Did the inputs and processes produce tangible items or outputs that can be quantified. Ask the group for examples of outputs…then show these examples.
Tuberculosis Technical Focus Area Activties and Update All-Staff Meeting 2006

10. WHAT DID THE PROGRAM ACHIEVE?
OUTPUTS
WHAT DID THE PROGRAM ACHIEVE?
Volume of drugs distributed to regional storage facilities
Number of primary health care centers implementing directly observed therapy
Number of laboratory technicians trained in smear microscopy
Number of TB cases tested for HIV
The results obtained at the program level through the execution of activities using program resources.
When we think of outputs, we ask – what did the program achieve? Did the inputs and processes produce tangible items or outputs that can be quantified. Ask the group for examples of outputs…then show these examples.
Tuberculosis Technical Focus Area Activties and Update All-Staff Meeting 2006

11. Outcomes WHAT ARE THE GOALS OF THE PROGRAM? Case Detection
Smear Conversion
Treatment Outcomes
Effects of a program at the target population level (for example, cohort analysis)
These outcomes are standardized for global reporting to WHO. They tell us the results of our efforts for the cohort of patients detected and treated by the NTP.
Tuberculosis Technical Focus Area Activties and Update All-Staff Meeting 2006

12. Medium and Long Term OUTCOMES
Outcomes for overall DOTS implementation
Case detection, Smear conversion, Treatment success
Medium term outcomes for NTP implementation
- Proportion of TB cases administered directly observed therapy
However, sometimes it is useful to think of outcomes in other ways.
Long vs. medium term outcomes – sometimes it is helpful to think of long term outcomes or the overall outcomes of DOTS implementation separately from outcomes we hope to achieve through all of the NTP’s activities, which may be observed in the short term and directly related to program outputs.
For example, long term outcomes may be the program results we hope to achieve with regards to case detection, smear conversion, and treatment outcomes, while medium term outcomes may be directly related to the implementation of one component of the DOTS strategy, for example, administration of directly observed therapy. We’ll give an example of this towards the end of the presentation to link this all together.
Tuberculosis Technical Focus Area Activties and Update All-Staff Meeting 2006

13. WHAT IS THE EFFECT OF THE PROGRAM ON THE EPIDEMIOLOGY OF TB?
IMPACT
WHAT IS THE EFFECT OF THE PROGRAM ON THE EPIDEMIOLOGY OF TB?
Transmission dynamics of TB
Incidence of pulmonary TB
Changes in mortality due to TB
Decrease in deaths due to HIV-related TB
Prevention of MDR-TB
Looking at the impact of our program answers this question: what is the effect of the program on the epidemiology of TB? AND most important, are you able to link these directly to your program efforts?
Show examples of impact – note that the epidemiology of TB is affected by many factors, not just our program, and this is one way we can distinguish outcome from impact. All of these items may be affected by other health programs or interventions or phenomena that are not addressed by TB programs.
Tuberculosis Technical Focus Area Activties and Update All-Staff Meeting 2006

14. Monitoring vs. Evaluation
Inputs, Process, Outputs, Outcomes:
Monitoring
Impact:
Evaluation
Here’s another way of thinking about it. These terms are related to the difference between monitoring and evaluation, covered in the first presentation.
Inputs, processes, outputs, outcomes are measured with monitoring activities and usually with data from your program and patients.
Impact is measured with evaluation activities and involves data sources beyond the routinely reported program and patient data
Tuberculosis Technical Focus Area Activties and Update All-Staff Meeting 2006

15. KEY TERMS – Directly Observed Therapy
INPUT – NTP guidelines for directly observed therapy (DOT)
PROCESS – NTP training workshops on how to implement DOT
OUTPUT – Number of nurses trained on administration of DOT
Intermediate OUTCOME – Proportion of smear positive TB cases on DOT increases
Long term OUTCOME – Treatment success rate increases to 85%
IMPACT – Decrease in morbidity and mortality due to TB and prevention of MDR -TB
Here is an example of how to apply these terms to NTP activities.
NTP manager wants to monitor and evaluate the implementation of directly observed therapy to smear positive TB cases.
What does he or she NEED to accomplish this goal? (CLICK)
Inputs – a supportive policy/guidelines developed and endorsed by the NTP and key partners.
What types of activities will he or she need to DO to accomplish this goal? (CLICK)
Process – Training workshops for doctors and nurses on how to implement DOT in the clinic
What will this ACHIEVE? (CLICK)
Outputs – The NTP will report on the number of nurses trained in administration of DOT.
How are these inputs, processes, and outputs linked to NTP GOALS? (CLICK)
Medium term OUTCOME – the proportion of smear positive TB cases receiving DOT will increase
Long term OUTCOME – the treatment success rate will increase to 85%
Finally, what is the effect of this NTP activity on the population? (CLICK)
IMPACT – Morbidity and mortality due to TB will decrease and we will prevent the emergence of MDR-TB.
Tuberculosis Technical Focus Area Activties and Update All-Staff Meeting 2006