1. Monitoring and Evaluation: A Review of Terms

2. Definition of a Goal
A goal is a statement, usually general and abstract, about a desired state, toward which a program is directed (Rossi & Freeman, 1993)
Goals are general statements describing an optimal situation (how things “should be”). For example, “All of our students should be fluent in at least two languages by the time they enter college.” The purpose of setting goals is to focus resources and actions toward desired outcomes.

3. What is a good goal?
To provide better treatment for people with tuberculosis in Country X
To achieve a treatment success rate of 85% in Country X by the end of 2006

4. Definition of an Objective
Specific and operationalized statements detailing the desired accomplishments of a program (Rossi & Freeman, 1993)

5. What are good objectives?
Increase the number of pharmacies with adequate supplies of drugs
Increase to 100% the proportion of pharmacies implementing NTP guidelines on drug management by the end of 2005

6. Objectives – Be SMART Specific Measurable Achievable Realistic
Time-Bound
note to trainer: useful to go through examples of each one of these components
Transition – now we will discuss specific program components and how they described in monitoring and evaluation terms.

7. INPUTS
A collection of resources (financial, personnel, equipment) which are the raw materials for the program

8. What do you need to implement the program?
Inputs
What do you need to implement the program?
Drugs and supplies
Laboratories
Nurses and doctors
DOTS strategy
Money

9. PROCESS
A set of activities in which program inputs are utilized in pursuit of the results expected from the program.

10. WHAT DOES THE PROGRAM DO?
Process
WHAT DOES THE PROGRAM DO?
Meetings
Training
Workshops
Reporting & Recording
Supervision
CEC-Removed Plans because they are thought of as inputs for some programs and thought it would be less confusing.
Notice, these can be concrete, tangible things or can be conceptual (supervision, for example). They are usually action-oriented, (for example, training or supervision activities that will produce something).

11. OUTPUTS
The results obtained at the program level through the execution of activities using program resources.
Usually, outputs are expressed as numbers and are taken from program data or documentation.

12. WHAT DID THE PROGRAM ACHIEVE?
OUTPUTS
WHAT DID THE PROGRAM ACHIEVE?
Volume of drugs distributed to regional storage facilities
Number of primary health care centers implementing directly observed therapy
Number of laboratory technicians trained in smear microscopy
Number of TB cases tested for HIV
When we think of outputs, we ask – what did the program achieve? Did the inputs and processes produce tangible items or outputs that can be quantified. Ask the group for examples of outputs…then show these examples.

13. OUTCOMES
Effects of a program at the target population level (for example, cohort analysis)
Participants are probably familiar with standardized outcomes that we measure in the target population, the TB patients that are detected and treated by the NTP.

14. Outcomes WHAT IS THE GOAL OF THE PROGRAM? Case Detection
Smear Conversion
Treatment Outcomes
These outcomes are standardized for global reporting to WHO. They tell us the results of our efforts for the cohort of patients detected and treated by the NTP.

15. Medium- and Long-Term OUTCOMES
Outcomes for overall DOTS implementation
Case detection, smear conversion, treatment success
Medium-term outcomes for NTP implementation
Proportion of TB-cases administered directly observed therapy
However, sometimes it is useful to think of outcomes in other ways.
Long vs. medium term outcomes – sometimes it is helpful to think of long term outcomes or the overall outcomes of DOTS implementation separately from outcomes we hope to achieve through all of the NTP’s activities, which may be observed in the short term and directly related to program outputs.
For example, long-term outcomes may be the program results we hope to achieve with regards to case detection, smear conversion, and treatment outcomes, while medium-term outcomes may be directly related to the implementation of one component of the DOTS strategy, for example, administration of directly observed therapy. We’ll give an example of this towards the end of the presentation to link this all together.

16. IMPACT
Population-level changes in the epidemiology of TB that are directly related to the program
Impacts are the results of our program efforts beyond the cases detected and treated by the NTP. We hope that our programs will have an effect at the population level, and when we speak of impact, this is the effect we are talking about.

17. WHAT IS THE EFFECT OF THE PROGRAM ON THE EPIDEMIOLOGY OF TB?
IMPACT
WHAT IS THE EFFECT OF THE PROGRAM ON THE EPIDEMIOLOGY OF TB?
Transmission dynamics of TB
Incidence of pulmonary TB
Changes in mortality due to TB
Decrease in deaths due to HIV-related TB
Prevention of MDR-TB
Looking at the impact of our program answers this question: what is the effect of the program on the epidemiology of TB? AND most important, are you able to link these directly to your program efforts?
Show examples of impact – note that the epidemiology of TB is affected by many factors, not just our program, and this is one way we can distinguish outcome from impact. All of these items may be affected by other health programs or interventions or phenomena that are not addressed by TB programs.

18. The Difference between Monitoring and Evaluation
Inputs, Process, Outputs, Outcomes:
Monitoring
Impact:
Evaluation
Here’s another way of thinking about it. These terms are related to the difference between monitoring and evaluation, covered in the first presentation.
Inputs, processes, outputs, outcomes are measured with monitoring activities and usually with data from your program and patients.
Impact is measured with evaluation activities and involves data sources beyond the routinely reported program and patient data

19. KEY TERMS - An Example
INPUT – NTP guidelines for directly observed therapy (DOT)
PROCESS – NTP training workshops on how to implement DOT
OUTPUT – Number of nurses trained on administration of DOT
Intermediate OUTCOME – Proportion of smear positive TB cases on DOT increases
Long term OUTCOME – Treatment success rate increases to 85%
IMPACT – Decrease in morbidity and mortality due to TB and prevention of MDR -TB
Here is an example of how to apply these terms to NTP activities.
NTP manager wants to monitor the implementation of directly observed therapy to smear-positive TB cases.
What does he or she NEED to accomplish this goal? (CLICK)
Inputs – a supportive policy/guidelines developed and endorsed by the NTP and key partners.
What types of activities will he or she need to DO to accomplish this goal? (CLICK)
Process – Training workshops for doctors and nurses on how to implement DOT in the clinic.
What will this ACHIEVE? (CLICK)
Outputs – The NTP will report on the number of nurses trained in administration of DOT.
How are these inputs, processes, and outputs linked to NTP GOALS? (CLICK)
Medium-term OUTCOME – the proportion of smear-positive TB cases receiving DOT will increase.
Long-term OUTCOME – the treatment success rate will increase to 85%.
Finally, what is the effect of this NTP activity on the population? (CLICK)
IMPACT – Morbidity and mortality due to TB will decrease and we will prevent the emergence of MDR-TB.

20. The “ONION” Chris Dye, 2002 estimated TB cases all true TB cases
cases presenting to health facilities
cases presenting to public health facilities
cases presenting to DOTS facilities
cases correctly diagnosed by DOTS facilities
diagnosed cases reported by DOTS facilities
This “onion”, developed by Chris Dye of the World Health Organization, is another way of illustrating how monitoring and evaluation differ, and why we need monitoring and evaluation to guide our programs.
With a good M&E system, will be able to track program successes for the core of the onion and the first or second layer through monitoring activities. The outermost layers require more than monitoring data – we will need an evaluation with rigorous methodology to determine if our program is having any effect at this level.
Chris Dye, 2002