IDEV 624 – Monitoring and Evaluation Assessing Program Theory Payson Center for International Development and Technology Transfer Tulane University

Monitoring and Evaluation (M&E): Definitions and Concepts

What is Monitoring and Evaluation (M&E)? Monitoring and evaluation of program activities provides the means for: –Learning from past experience –Improving service delivery –Planning and allocating resources –Demonstrating results as part of accountability to stakeholders (World Bank)

Monitoring: –A continuing function that uses systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing development intervention with indications of the extent of progress and achievement of objectives and progress in the use of allocated funds Evaluation: –The systematic and objective assessment of an on-going or completed project, program or policy, its design, implementation and results –The aim is to determine the relevance and fulfillment of objectives, development efficiency, effectiveness, impact and sustainability (OECD, Glossary of Key Terms in Evaluation and Results Based Management, 2002)

6/5/2016 What is the problem? Situation Analysis & Surveillance What are the contributing factors? Determinants Research What interventions and resources are needed? Needs, Resource, Response Analysis & Input Monitoring What interventions can work (efficacy & effectiveness)? Efficacy & Effectiveness Studies, Formative & Summative Evaluation, Research Synthesis Are we implementing the program as planned? Outputs Monitoring What are we doing? Are we doing it right? Process Monitoring & Evaluation, Quality Assessments Are interventions working/making a difference? Outcome Evaluation Studies Are collective efforts being implemented on a large enough scale to impact the epidemic? (coverage; impact)? Surveys & Surveillance Understanding Potential Responses Monitoring & Evaluating National Programs Determining Collective Effectiveness ACTIVITIES OUTPUTS INPUTS OUTCOMES & IMPACTS A Public Health Questions Approach to HIV/AIDS M&E Are we doing the right things? Are we doing them right? Are we doing them on a large enough scale? Problem Identification (UNAIDS 2008)

6/5/2016 Most Some Few* All Input/ Output Monitoring Process Evaluation Outcome Monitoring / Evaluation Levels of Monitoring & Evaluation Effort Number of Projects * Disease impact monitoring is synonymous with disease surveillance and should be part of all national-level efforts, but cannot be easily linked to specific projects Strategic Planning for M&E: Setting Realistic Expectations 6 Impact Monitoring / Evaluation

(World Bank 2009)

Assessment of Program Theory

Program theory: –Conception of what must be done to bring about the intended social benefits Assessment of program theory: –Task of assessing the quality of program conceptualization Good: If it represents the “know-how” necessary to attain the desired results Poor: If it would not produce the intended effects even if the project is implemented well  Evaluators find that program theory often is not very convincing (Rossi/Lipsey/Freeman: 2004)

Program Impact vs. Program Process Theory Program process theory: –Represents assumptions about the capability of the program to provide services that are accessible to the target population and compatible with their needs Program impact theory: –Involves the sequence of casual links between program services and outcomes that improve the targeted social conditions (Rossi/Lipsey/Freeman: 2004)

Evaluability Assessment Evaluability assessment: Does the program/project meet the minimal conditions for evaluation? 1.Description of the program model (the program’s design and logic) with particular attention to defining goals and objectives 2.Assessment how well-defined and evaluable that model is 3.Identification of stakeholder interest in evaluation and the likely use of findings  A common outcome is that managers and sponsors recognize the need to modify the program (Rossi/Lipsey/Freeman: 2004)

Assessment of Program Theory (cont.) Program theory can be “implicit” or (better) “articulated” If implicit, the evaluator must extract and describe program theory before a program can be evaluated –Objective: To depict the program “as intended”, based on the actual expectations held by decision-makers about what the program is supposed to do and the results expected to follow Program theory has to be assessed in relationship to the needs of the target population that the program is intended to serve (  Needs assessment) (Rossi/Lipsey/Freeman: 2004)

Theory of Change: Theory of how an initiative leads to desired results (World Bank 2009)

Standard Flow Chart

(World Bank 2009) Example: Theory of Change

Logic Model - Example (for a teen mother parenting program) (Rossi, P. H. et al., p. 95)

The Management Plan

Key Steps 1.Conduct a needs assessment 2.Develop and refine a conceptual framework 3.Create a management plan: –Clarify your goals –Define your objectives –Identify your key activities

Initial Conceptual Model Target Condition Factor Activity

The Management Plan: Goals Objectives Activities

Program Management Plan

Management Plan Structure Goal –Objective 1 Activity 1 Activity 2 Activity 3 –Objective 2 Activity 1 Activity 2 Activity 3 –Objective 3 Activity 1 Activity 2 Activity 3

Goals The goal is a "general summary of the desired state that a project is working to achieve" (Margoluis/Salafsky 1998: 60). An ideal goal is –visionary –relatively general –brief –measurable Typically, a project has only one goal.

Develop Goals Reshape target condition (conceptual model) into a goal: –Future oriented –Positive statement –Brief and clearly articulated –Measurable

Develop Goals (cont.) Refer to the social problem that you are going to address Refer to the target population and location Use clear terminology

Poorly Written Goal Project goal: Increase knowledge about sexual and reproductive health in Mexico –What is wrong?

Poorly Written Goal Project goal: Increase knowledge about sexual and reproductive health in Mexico –What is wrong? Goal does not refer to the major health or social problem to be addressed –Why is it important to increase knowledge? –What do we think will happen if knowledge is increased? Goal does not provide information about the focus population; it is to general

Revised Goal Project goal: Improve the sexual and reproductive health of young people ages 15 to 24 in City X, through the reduction of unwanted pregnancy and sexually transmitted infections, including HIV

More Project Goal Examples Well-written goals: –To reduce incidence of HIV infection among young people ages 15 to 24 in Country X. –Increase the capacity of youth-serving organizations to implement more effective peer education programs for preventing HIV infection among sexually active young people (ages 15-24).

Objectives The objectives are "specific statements detailing the desired accomplishments or outcomes of a project. If a project is well conceptualized and designed, realization of a project's objectives should lead to the fulfillment of the project's goal" (Margoluis/Salafsky 1998: 64).

Goals Versus Objectives Goals are usually aspiring and far-reaching – not limited in terms of time or other measurement Objectives should be measurable, quantifiable, time-limited and action-oriented  The test of a goal is to make certain that it fits within your organization’s mission; the test for objectives is that they lead to the accomplishment of a goal

Develop Objectives Select a factor (threat) from your conceptual model Develop 1-3 specific objectives related to this factor –What do we need to do to accomplish our desired outcome? –What information do we need to demonstrate we actually accomplished what we planned to do?

Initial Conceptual Model Target Condition Factor Activity

Develop Objectives (cont.) It is important that objectives be realistic, not just impressive, as lofty, unfounded objectives undercut the credibility of the entire project Well-written objectives identify: –WHO will be reached –WHAT change will be achieved –IN WHAT TIME PERIOD the change will be achieved –WHERE (in what location)

SMART Objectives Objectives should be “SMART” –Specific To avoid differing interpretations –Measurable To monitor & evaluate progress (preferably quantitatively) –Appropriate To the problems, goal and your organization’s mission –Realistic Achievable, yet challenging and meaningful –Time-bound With a specific time for achieving them

Objectives and Action When writing objectives, choose action verbs that indicate a change and the direction that change will take Examples: –Decrease –Increase –Strengthen –Improve –Enhance

Poorly Written Objective Project objective: Train 60 peer educators to promote the ability to practice safer sex –What is wrong?

Poorly Written Objective Project objective: Train 60 peer educators to promote the ability to practice safer sex –What is wrong? This objective refers to a strategy or activity –Why does the organization want to train peer educators – what change is sought? Objective is not time-bound Focus population is not specified –For this project, is the change to be achieved among the peer educators or among the people the educators will reach? Unclear terminology –What does the ability to practice safe sex mean? How will the organization recognize it when it occurs?

Revised Objective Project objective: At the end of the three-year project, increase condom use among sexually active young people (ages 15 to 24) in Community X

More Sample Objectives By the end of the first project year, increase knowledge and skills of staff in five clinics in City X in how to counsel young clients (ages 15 to 24) about HIV prevention, with emphasis on condom use and negotiation. By the end of the two-year project, increase the availability of information about the effectiveness of community-based versus center-based peer education programs in increasing condom use among sexually active youth (ages 15 to 24).

Activities The activities are "specific actions or tasks undertaken by project staff designed to reach each of the project's objectives" (Margoluis/Salafsky 1998: 73). The project activities should be –linked –focused –feasible –appropriate

Feasibility Available resources Constraints Cost-effectiveness Any better alternatives?

Analysis of Alternatives There is no fixed format for the analysis of alternatives. The purpose of this tool is to systematically compare various possible intervention strategies focusing on the project's key criteria. The analysis of alternatives helps to choose the best approach for any intervention. This may take a form similar to the table below. Criteria 1 (e.g. social) Criteria 2 (e.g. environment) Criteria 3 (e.g. financial) Strategy 1 Strategy 2 Strategy 3

Develop Activities Why do this activity? - Justification How will activity be carried out? Who is responsible? When will activity take place? Where will the activity take place? Underlying assumptions Prerequisites

Management Plan Structure Goal –Objective 1 Activity 1 Activity 2 Activity 3 –Objective 2 Activity 1 Activity 2 Activity 3 –Objective 3 Activity 1 Activity 2 Activity 3

Management Plan Timeline

Timeline A bar graph which lists the major activities and tasks involved in the project. Shows how long the activities are supposed to last and the relationship between different activities.

Management Plan Timeline

Gantt Chart Visualizes tasks and milestones of a project over time. Bars typically represent tasks and diamonds represent milestones. Reflects the length of tasks. May indicate the exact dates, the persons in charge, etc. for each project component. Widely used by project managers in the project planning and implementation.

PERT Chart Program Evaluation Review Technique (PERT chart). Visualizes the project with all of its elements over time. Rectangles typically represent events or milestones and arrows represent the sequential tasks. Also known as Critical Path Method (CPM).

Insert Activities into Conceptual Model

Basic Model Target Condition (Goal) Factor (Objective) Factor (Objective) Factor (Objective) Factor (Objective) Factor Activity

Margoluis and Salafsky p78

Single Objective 3.1 Conduct community meetings 3.2 Train herders in fire management 3.3 Show herders park boundaries Target Condition: Grassland & Savannah Ecosystems Grassland & Savannah Dynamics Obj 3: Eliminate FIRE FIRE Drought Cattle Grazing in Park

Finalize Management Plan