Monitoring and Evaluation PISA INSET Training Corporate slide master With guidelines for corporate presentations Module 4 Monitoring and Evaluation 1 Module 3 - An Introduction to Thinking Skills 1

The project cycle ASSESSMENT EVALUATION Monitoring PROGRAMME DESIGN Disaster The project cycle ASSESSMENT EVALUATION Monitoring PROGRAMME DESIGN IMPLEMENTATION source: IFRC 3Ps/SPHERE 18-Sep-18

Definition Monitoring It is NOT only about PROCESS ‘The systematic and continuous assessment of the progress of a piece of work over time….’ ‘To continuously measure progress against programme objectives and check on relevance of the programme’ It involves collecting and analysing data/information It is NOT only about PROCESS There are various definitions but they all aim at the same: we measure progress and changes need to be made if progress is not made. Obviously ‘progress’ has all to do with what our programme originally aimed for, i.e. the objectives. Are we in the process of reaching them? If not, we need to make adjustments to ensure we reach these objectives eventually. ‘The systematic and continuous assessment of the progress of a piece of work over time….It is a basic and universal management tool for identifying the strengths and weaknesses in a programme. Its purpose is to help all the people involved make appropriate and timely decisions that will improve the quality of the work.’ Gosling and Edwards, 1995 cited in ALNAP Review of Humanitarian Action 2003 *

Purpose of monitoring to document progress and results of project to provide the necessary information to Management for timely decision taking and corrective action (if necessary) to promote accountability* to all stakeholders of a project (to beneficiaries, donors, etc) Source: MTV for UCL + Makerere University School of Public Health, Uganda *accountable = being responsible to someone for some action Monitoring and evaluation, though two distinct activities are very closely linked. Monitoring is a routine activity with data collected on a regular e.g. daily or monthly basis. Its basic purpose is to keep track of programme activities and improve the efficiency of interventions. Evaluation tends to be episodic, undertaken at critical points in a project cycle and its basic purpose is more to do with improving effectiveness and informing future programming. Monitoring data provides essential inputs into more episodic evaluation. Monitoring data may highlight specific issues in the programme’s implementation that require deeper investigation through evaluation to be resolved. In turn, evaluation can help to identify what needs to be monitored in the future. In a well designed M&E system, data routinely collected through monitoring activities can contribute greatly towards evaluation *

Information collected for monitoring must be: Useful and relevant Accurate Regular Acted upon Shared Timely Source: MTV for UCL + Makerere University School of Public Health, Uganda *

Definitions Evaluation It involves the objective assessment of an The aim is to determine relevance and fulfilment of objectives, as well as efficiency, effectiveness, impact and sustainability of a project. It involves the objective assessment of an ongoing or completed project/programme, its design, implementation and results. These are all terms we will look at more in detail. Evaluations are also very important to identify LESSONS LEARNT *

Evaluations are done when / because: Monitoring highlights unexpected results More information is needed for decision making Implementation problems or unmet needs are identified Issues of sustainability, cost effectiveness or relevance arise Recommendations for actions to improve performance are needed Lessons learning are necessary for future activities Additionally, the donor (often from one or various Governments) that funds the project is accountable to its tax payers, and the donor wants to know what exactly is done with the money and whether the objectives were reached. And if not, why not.

To evaluate projects well is a real skill! And you often need a team… Source: UCL + Makerere University School of Public Health, Uganda *

M&E in emergencies? YES Any project without Monitoring and/or Evaluation is a BAD project Source: MTV for UCL + Makerere University School of Public Health, Uganda *

Examples of data collection methods for M&E Quantitative Methods Qualitative methods Administering structured oral or written interviews with closed questions Semi structured interviews e.g. key informant Population based surveys Focus group discussion Reviewing medical and financial records Observing Completing forms and tally sheets Case studies Direct measurement (anthropometry, biochemical analysis, clinical signs) Mapping, ranking, scoring Lot quality assessment Problem sorting, ranking

Indicators An indicator is a measure that is used to show change in a situation, or the progress in/results of an activity, project, or programme. Indicators: enable us to be “watchdogs”; are essential instruments for monitoring and evaluation. are objectively verifiable measurements Source: UCL + Makerere University School of Public Health, Uganda

What are the Qualities of a Good Indicator? Specific Measurable Achievable Relevant Time-bound The Sphere Project provides the most accepted indicators for nutrition and food security interventions in emergencies: see Module 21. Source: MTV for UCL + Makerere University School of Public Health, Uganda What are the characteristics of a good indicator? A good indicator is SMART: S for SPECIFIC: By specific, we mean that it specifies the magnitude of the attribute that we’re measuring in a particular time frame and for a particular population In other words, that they measure what they’re supposed to measure… blood retinol measures vitamin A status (not iron status, not any other vitamin…) Can you think of another word starting with “S” that describes a good indicator? Good! it’s SIMPLE: clearly and precisely defined M for Measurable: Measurable indicators are objective: we are able to quantify them You can measure the temperature, you can count how many correct answers students get on a test, you can observe if someone is washing their hands before they handle food Can you think of other examples of indicators and how to measure them? A for Achievable: Achievable indicators means that we can actually obtain them: you need to have the resources and capacities to collect, store and process the information obtained from that indicator R is for relevant: Relevant to you project, your situation And T is for Time bound: Always stating the time period in which that indicator was measured Beginning and end of projects are commonly used time-marks Seasons and seasonality too: amount of grain produced in the month of July, total sales in the pre-holidays season, malnutrition rates in the lean (or hungry) season preceding the harvest. Can you think of other examples of indicators that are time-bound? SMART Initiative[1] The Standardised Monitoring and Assessment in Relief and Transition (SMART) Initiative - is an interagency initiative, begun in 2002, to improve the M&E of humanitarian assistance interventions through: • The development of standardised methodologies for determining comparative needs based on nutritional status, mortality rate and food security. • Establishing comprehensive, collaborative systems to ensure reliable data is used for decision-making and reporting A Standardised Training Package (STP) for SMART methodology has recently been released. [1] More information available at http://www.smartmethodology.org/ And there is also the SMART initiative…. Standardised Monitoring and Assessment in Relief and Transition Initiative - interagency initiative to improve the M&E of humanitarian assistance

Types of indicators Examples? Indicators exist in many different forms: Examples? Direct Direct indicators correspond precisely to results at any performance level. Indirect or "proxy" indicators demonstrate the change or results if direct measures are not feasible. Indirect / proxy Indicators are usually quantitative measures, expressed as percentage or share, as a rate, etc. Indicators may also be qualitative observations. Qualitative Quantitative Source: UCL + Makerere University School of Public Health, Uganda Direct - Number of children with acute malnutrition Proxy – diet composition for nutritional status Global / standardised Standardised global indicators are comparable in all settings. Other indicators tend to be context specific and must be developed locally. Locally developed

Impact Outcome Output Input Source: UCL + Makerere University School of Public Health, Uganda Input

Impact Outcome Output Input Related to Goal Related to Objectives (or Purposes) Output Related to Outputs Source: UCL + Makerere University School of Public Health, Uganda Input Related to Activities/Resources

Impact Outcome Output Input * Malnutrition rates amongst young children reduced Related to Goal Outcome % of young children getting appropriate complementary food Related to Objectives (or Purposes) X number of mothers know about good complementary food and how to prepare that Output Related to Outputs Source: UCL + Makerere University School of Public Health, Uganda Nutritional education to mothers on complementary food Input Related to Activities/Resources *

What is a Log Frame? The logical framework or logframe is an analytical tool used to plan, monitor, and evaluate projects. ? ? ? The logical framework or logframe derives its name from the logical linkages set out by the planner(s) to connect a project’s means with its ends. ? Victim of a log frame?

Log Frames IMPACT OUTCOME INPUTS Log frames appear in different shapes and with different names sometimes, but overall they look like this. INPUTS

PART 1: What’s the issue you want to solve? Using this space, please respond to the following: - Write about your team’s topic. - Explain why your team selected this topic. - Explain what issue you will address in your action plan.

PART 2: What’s your action plan? Using this space, please respond to the following: What are the step-by-step actions your team will take to address the environmental issue? List the specific responsibilities of each team member toward achieving your plan. How will your team measure success? How will you quantify impact? (e.g. tallying how many gallons of water were saved, by tracking how many cans were recycled.)

PART 2: What’s your action plan? (continued)

PART 3: How did you implement your ideas? Using this space, please respond to the following: What did your team do? Did you organize a school event or a community wide event to raise awareness of the issue? Tell us about it. Did you work with any local environmental groups, governmental agencies, schools, etc? Tell us about it. What did you do to get the word out about your project? (e.g. create a Web site, TeacherTube video, press release, write a blog, etc.) Please provide a list of any internet links to your website, online videos, blog etc.

PART 3: How did you implement your ideas? (continued)

PART 4: Evaluate your plan. What are the results? Using this space, please respond to the following: In what ways was your plan successful? What challenges did you face? How did you raise awareness in you school communicate as well as the greater community? What were the quantifiable results? What did you learn from this experience?

PART 4: Evaluate your plan. What are the results? (continued)