1. Session 3: Principles of Monitoring and Evaluation
Principles of M&E
Session 3: Principles of Monitoring and Evaluation
[MATERIALS FOR THE MODULE:
HANDOUTS OF THE SLIDES, AS NECESSARY
LARGE PIECES OF POSTER BOARD OR FLIP CHART PAPER, ONE PER GROUP, FOR ACTIVITY
THICK MARKING PENS, ONE OR TWO PER GROUP.]
Now that we have a greater understanding of M&E plan components and how to create strong goals and objectives, let’s move on to fundamental monitoring and evaluating concepts. This course is not a comprehensive course to teach you everything you need to know about monitoring and evaluation; however, it is important that we all understand the basics of M&E so that we can effectively support M&E efforts for our programs.
SP - 16 July 07 - Pretoria

2. Principles of M&E
Session Overview
Purposes of M&E Definitions of monitoring and evaluation Definitions of program components Developing program frameworks
This session establishes the foundation for understanding M&E, including the reasons why we develop M&E plans and systems, the differences between monitoring and evaluation; and which questions each of these can answer; how to break down your program into program components and how to explain these program components through a program framework.
SP - 16 July 07 - Pretoria

3. Session Learning Objectives
Principles of M&E
Session Learning Objectives
By the end of the session, the participant will be able to:
explain the main purposes of M&E;
understand the difference between monitoring and evaluation; and
know the different program components and how to develop a framework with these components.
[READ SLIDE]
SP - 16 July 07 - Pretoria

4. Purposes of M&E Improve programs by identifying the aspects that:
Principles of M&E
Purposes of M&E
Improve programs by identifying the aspects that:
are working according to plan, and
those that need to be modified.
Track changes in the interventions and in the desired outcomes.
Reach informed decisions regarding the effective and efficient use of program resources.
API programs operate in a rapidly evolving environment, in which the target species for interventions and target interventions themselves are constantly in motion. As we saw with the emergence of the H1N1 pandemic influenza, our program efforts are continually subject to new demands. In addition, countries are dealing with a host of other urgent health priorities. In this environment, M&E is critical to our success. M&E helps us enhance and document our program performance. The purposes of M&E are to: [READ SLIDE]
#1:
What happened in the past and what is happening now? Is the program on track? Does the program need to change?
#2:
Where are the successes? Where are we making progress?
#3:
Are the program’s resources used effectively and efficiently? If resources are declining, how can we target the remaining resources effectively?
SP - 16 July 07 - Pretoria

5. Principles of M&E
Purposes of M&E
Strengthen program design and improve program implementation.
Generate knowledge, such as:
Identifying factors (individual, community, programmatic) that influence health outcomes.
Justify the use of allocated resources.
Meet an organizational requirement.
What are some other purposes of M&E? [READ SLIDE]
There is an urgent need for our field, including decision makers, researchers, and advocates, to demonstrate the importance of investment in an integrated approach to health that encompasses both the animal and human health sectors and to make a stronger case for increased investment in this field. In addition, M&E identifies “lessons learned” to help improve our future efforts.
[FOR THE LAST ITEM ON THE SLIDE, “MEET AN ORGANIZATIONAL REQUIREMENT,” ASK PARTICIPANTS WHAT KINDS OF ORGANIZATIONAL REQUIREMENTS THEY CAN THINK OF – EXAMPLES: REPORTING TO THE FUNDER, REPORTING TO NATIONAL AUTHORITIES].
SP - 16 July 07 - Pretoria

6. Key M&E Questions Did the program achieve its objectives?
Principles of M&E
Key M&E Questions
Did the program achieve its objectives?
Did the target population benefit from the program? At what cost?
Can improved health outcomes be attributed to program efforts?
Which program activities were more (or less) important/effective?
Here are some key questions that can be answered when effective systems for collecting M&E data are in place. [READ SLIDE]
SP - 16 July 07 - Pretoria

7. What is Monitoring? Monitoring involves:
Principles of M&E
What is Monitoring?
Monitoring involves:
routine tracking of information about a program and its intended outputs, outcomes, and impacts;
measurement of progress toward achieving program objectives;
tracking costs and program/project functioning;
providing a basis for program evaluation when linked to a specific program.
Let’s look at the definitions for monitoring and evaluation.
Monitoring provides regular feedback to compare what was planned to actual events by tracking: costs; personnel; implementation time; organizational development; and economic and financial results.
SP - 16 July 07 - Pretoria

8. What is Evaluation? Evaluation may involve:
Principles of M&E
What is Evaluation?
Evaluation may involve:
activities designed to determine the value or worth of a specific program;
the use of special study designs or special studies to examine program effectiveness;
the examination of performance against defined standards, an assessment of actual and expected results, or the identification of relevant lessons.
Evaluation helps us answer questions such as: How can we tell if our program has made a difference, whether we have accomplished our objectives and whether an intervention works? Special study designs help us answer these types of questions and often involve data collection in addition to the data collected as part of routine monitoring. We will return to evaluation designs in session 7. The key point for this session is to understand that evaluation is related to but distinct from routine monitoring.
SP - 16 July 07 - Pretoria

9. Comparing M&E Monitoring: What are we doing?
Principles of M&E
Comparing M&E
Monitoring: What are we doing?
Tracking inputs and outputs to assess whether programs are performing according to plans
(e.g. people trained, outbreaks investigated)
Evaluation: What have we achieved?
Assessment of impact of the program on behavior or health outcome
(e.g. reporting sick poultry, AI case fatality ratio)
Monitoring is the routine tracking of priority information about a program and its intended effects.
Evaluation is a collection of activities designed to determine the value or worth of a specific program, intervention, or project. That means being able to link a particular output or outcome directly to a particular intervention.
Monitoring answers the question, “What are we doing?” by looking at routine data to track:
Inputs
Activities
Outputs (activities completed)
Budget
Evaluations answer the question, “What have we achieved?” by periodically measuring progress towards:
Program objectives (outcomes/impact); are changes attributable to program activities?
Monitoring is routine. It tell us what we are doing by tracking inputs and outputs.
Evaluation is an episodic assessment of achievement.
SP - 16 July 07 - Pretoria

10. Logical Framework personnel trained; media materials produced; or
Definition:
Diagrams that identify and illustrate the linear relationships flowing from program inputs, processes, outputs, outcomes and impacts.
Inputs or resources affecting processes or activities that produce immediate results or outputs, ultimately leading to longer term or broader results, or outcomes.
Purpose:
Provides a streamlined interpretation of planned use of resources and desired ends.
Clarifies project/program assumptions about linear relationships between key factors relevant to desired ends.
Other terms used include M&E frameworks, indicator matrixes.
API programs are complex and involve many different activities. We can link these together through a logical framework. We looked at logical frameworks briefly in session 2. As we discussed then, logical frameworks are diagrams that identify and illustrate the linear relationships flowing from program inputs, through processes, outputs, and leading to outcomes. Inputs, or the program’s resources, provide the fuel for processes, or the program’s activities. These processes produce immediate or direct outputs or effects; in other words, the program’s activities yield some intermediate results. These outputs lead to long-term or broader, overall results, or the program’s outcomes, which lead to impacts. Impacts are generally achieved in collaboration with many other organizations working towards the same vision.
The following are examples of typical inputs for an API-focused program:
Government and other donors
Veterinary, health personnel
Protocols and guidance
Training materials
Personal Protective Equipment
Some typical activities for these programs are:
Outreach
Materials distribution
Workshops
Training
Examples of outputs include number of:
live bird market assessments completed;
biosecurity recommendations implemented;
personnel trained;
media materials produced; or
outbreak investigations conducted
Examples of outcomes could include changes in:
Knowledge or skills
Attitudes and beliefs
Behaviors
Access
Policies
Environmental conditions
Examples of impact include changes in:
AI infection rates
AI morbidity
AI mortality
A logical framework is a useful tool for identifying and illustrating operational factors important to achieving a successful ultimate outcome. For this reason, such a framework is sometimes perceived as the most useful tool for program monitoring. Since only the framework specifies inputs, it is also the only model that can directly point toward indicators for process monitoring. One of its strengths is that it can help identify linkages and key processes, highlighting the relationship between resource allocation and success, as well as where problems may lie if goals are not being achieved.

11. Key Terms
Inputs: These may include human, financial, social, political, or other resources needed to begin and complete the program.
Processes: Activities that produce immediate results – What will you be doing?
Outputs: Involve immediate results from your processes – What are the results of your activities?
Outcomes: Longer term or intermediate results – The progress towards achieving your objectives.
Impact: Long-term changes necessary to sustain your goal.
Although you may use other definitions, for the purposes of this course and MEASURE Evaluation, these are the definitions of these key terms. [READ SLIDE]. Generally, in M&E within our API programs, we are concerned most with measuring outputs and outcomes. Impact is much harder and expensive to measure and often too long term for the funding cycles in which we operate.

12. Components of an Avian Influenza Program
Principles of M&E
Components of an Avian Influenza Program
Population
level
Program level
Outcomes
Inputs
Processes
Outputs
Resources
Staff
Drugs
Supplies
Equipment
Functions,
Activities
Training
Logistics
IEC
Intermediate
Increased knowledge,
behavior of ways to protect
poultry from AI
Long-term
Infection rate
Mortality
Services
% facilities providing quality
Infection control
% backyard farmers
receiving AI awareness materials
# trained staff
Utilization
% commercial
farmers vaccinating flocks
This diagram shows the typical elements at each stage of the logical framework and the links between these elements to achieve the ultimate goals and objectives of the program. These types of frameworks assist program managers and M&E planners as they clarify the direct relationships among different elements. It also enables monitoring at each stage of the program model. In the event that progress is not being made, it can help to identify where in this framework the project is not achieving – Are there insufficient inputs? Are activities not taking place? Are the activities not having the intended effect? Even if they are having the intended short-term effect, is this not resulting in changes in longer-term outcomes?
Note that the inputs, processes, and output levels are generally measured through program data. However, outcomes and impacts are measured at the population level through surveys or disease surveillance, or other large-scale data collection methods.
SP - 16 July 07 - Pretoria

13. Logical Framework: Training Activity
INPUT
PROCESS
OUTPUT OUTCOME IMPACT
Time and skills to develop new biosecurity training curriculum for market owners
Conduct
training events
Market owners trained in biosecurity methods
Improvement in market conditions
Reduction in HPAI/H5N1 virus circulation
Here’s another example using a specific activity. Let’s talk through the arrows as part of the process of creating the model.
This logic model presents a straightforward view of a project component designed to reduce HPAI/H5N1 virus circulation through training market owners in biosecurity methods. As you can see, it does not try to account for all of the factors that may be influencing operation and results, but instead focuses on the project’s immediate activities and effects of those activities. This narrow focus assists program managers and M&E planners as they clarify the direct relationships among elements of particular interest within a particular program effort.
In this example, using arrows, you can talk through how you think your input will influence your process, which will affect your output, which will lead to your outcome and eventual impact. It operates in an “if – then” format. For example, we could say: “If we put in the time and skills to develop a biosecurity training curriculum, then, we could conduct training events with the curriculum (the process, or activity). If we conduct those processes/activities, then the market owners will be trained in these techniques – the output, or immediate result of the activity. If they are trained, then they will be able to improve the conditions in the market using these biosecurity methods – the short-term change in practice as a result of our program. If they improve market conditions in the short term, then in the long term (impact), they will reduce the circulation of HPAI/H5N1.
Make sense? Any questions?

14. Small Group Activity Return to small groups
Return to the program you described to:
develop a logical framework for at least three activities/processes of your program; and
work through inputs, processes (activities), outputs, outcomes, and impact.
After about 30 minutes, we will reassemble and a member of each small group will share one logical framework.
Return to your small groups for these framework activities. Now you will practice developing your logical framework. Select three activities from your project. Then, using a series of “if-then” statements, create a logic model for each activity with at least one input, process, output, outcome, and impact. You will have about 30 minutes for this activity. Then, each group will share one logical framework with the larger group. I will let you know when 10 minutes remain for the small-group work. At the end of this activity, you should have one logical framework written out on the large paper to be put on the wall. As you are completing this activity, be sure to refer back to your vision, goals, and objectives.
[MONITOR THE GROUPS TO MAKE SURE THEY ARE PROGRESSING AND TO ANSWER QUESTIONS.]
[RECOMMENDED MATERIALS FOR THIS ACTIVITY ARE:
COPIES OF THE POWERPOINT SLIDE “BLANK LOGIC MODEL” FOR EACH INDIVIDUAL OR GROUP
LARGE PIECE OF POSTER BOARD OR FLIP CHART PAPER, ONE PER GROUP
THICK MARKING PENS, EACH GROUP SHOULD HAVE PENS FROM THE PREVIOUS ACTIVITY.]

15. Small Group Projects
Present your findings to the group.