Monitoring and Evaluation Dr. Raghu Bista Anil Kumar Gupta

Outline Basics of M&E framework Result orientation in practice: Monitoring tools, techniques and issues in selected projects

Implementation of Project/Programs Once preparation is completed, finance is assured and design is finalized, project/program is forwarded for implementation. During implementation phase, Project/program is put into action. Roads will be contracted and built, project activities will be started and completed, project output will be produced. So this is the investment phase of the project. Here, inputs are converted into outputs

Outputs of Project/Programs Implementation The output of the implementation phase is the new road, new hydropower capacity, new bridge, new hospital department, new irrigation system, trained manpower, enhanced capacity, new product etc.

Project Implementation To create expected outputs, projects are divided into work packages and activities, activity time and activity cost are estimated, time schedules are prepared. Likewise budget, resources and procurement plan are efficiently utilized

Project Control During project/program implementation, control is management function to corrective action to address gap between and actual and planned performance. Clearly, the need to exert proper control mandates the need to monitor the proper activities and elements of the project

The Project Control Cycle 1. Setting a Goal 4. Taking Corrective Actions 2. Measuring Progress 3. Comparing Actual With Planned

Monitoring “Monitoring is the collection, recording, and reporting of project information that is of importance to the project manager and other relevant stakeholders” (Mantel, Meridith, Shafer, Sutton & Gopalan, 2005). Monitoring is a continuous function that uses systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing development intervention with indications of the extent of progress and achievement of objectives and progress in the use of allocated funds. (World Bank, 2011)

Purpose of Monitoring To indicate at the earliest instance any shortcomings with regard to achieving intended objectives so that measures can be undertaken in good time To monitor the development of the project as a whole, and its component projects, in relation to changes in the context and circumstances of their implementation. To implement a rapid problem identification system as well as a system for internal communications to the various stakeholders.

Purpose of Monitoring Cont.. To provide forecasts on cost, To forecast resources and schedule To facilitate evaluation process during and after activities, through the definition of specific indicators.

Process of Monitoring Identification of Indicators Collection of information Analysis of information Use of information for Action

Work Schedule and Monitoring

Project S-Curve as Tool for Cost Monitoring Project S curve consists of presenting costs against time for both budgeted and actual expenditure.

The Project S-Curve Cumulative Cost ($ in thousands) Elapsed Time (in weeks) 10 5 45 40 35 30 25 20 15 50 60 $20,000 Negative Var Cumulative Budgeted Cost Cumulative Actual Cost

Evaluation Evaluation can be defined as a process by which general judgments about quality, goal attainment, project/program effectiveness, impact and costs can be determined. It can therefore be conceptualized as a review of the whole project/program in order to assess its overall value and effects. Evaluation is a systematic and purposeful undertaking carried out by internal or external evaluators to appraise relevance, effectiveness, efficiency, impacts as well as sustainability generated by the policies, plans, programmes, and projects under/after implementation. (National Planning Commission, 2013)

Evaluation at Different Level Evaluation is also classified by subject such as project level, program level, sector Level, policy level.

Evaluation Type Ex-ante Evaluation Mid Term Evaluation Completion Evaluation Ex-post Evaluation (Impact Evaluation)

M & E During Programme/Project Period (National Planning Commission , 2013)

Implementation of projects Type of Evaluation Start of Implementation Completion of Implementation 2. Implementation of projects 4. 1. 3.

Ideal situation-M& E Complementary Monitoring Evaluation

Difference Between M & E Evaluation is assessing the entire project cycle. Evaluation looks for lessons to be learned from both success and lack of success, and also looks for best practices which can be applied elsewhere. It is concerned with the assessment of progress against predetermined goals, outcomes and their effects and impacts on stakeholders. It is generally conducted by an independent outsider Monitoring is a continuing function that takes place throughout the implementation of a project/programme It focuses on the implementation of the project comparing what is delivered with what is planned It is directly related with inputs, processes and outputs. It is generally conducted by the implementing agency.

Result based M& E Video

Result Based Management Result based management is perspective that focus on achievement of development results. It is a management strategy that focuses on the processes, products and services contributing to the achievement of development results (NPC, 2013)

How the Results Come Once a project is designed and implemented, it will produce outputs (For example 25 trained math teachers) The trained math teachers will use improved methods to teach, the result will be increased math solving competency of students In exam, number of students to pass exam will be increased with better marks as ultimate results and will be able to get scholarship

Results Chain Impacts Activities Inputs Outputs Outcomes

The Power of Measuring Result If you do not measure results, you cannot tell success from failure If you cannot see success, you cannot reward it If you cannot reward success, you are probably rewarding failure If you cannot see success, you cannot learn from it If you cannot recognize failure, you cannot correct it If you can demonstrate results, you can win public support (Osborne & Gaebler 1992)

Result Base M & E Results-based monitoring is a process of incessantly collecting and analyzing information on the extent to which results from the implementation of a given plan, policy, programme or project have been achieved so as to compare it with the expectations. Results-based Evaluation is an undertaking that appraises relevance, efficiency, effectiveness, and sustainability of a given plan, policy, programme or project by making comparison of the outcomes of implementation with the intended results. (NPC, 2013)

Essence of Result Based M & E Traditional M & E is more focused on completion of assignment/program/policy Result Based M & E gives emphasis on achievement of ultimate results from the intervention So it focus on continuous monitoring and evaluation of impacts during project implementation and after project implementation to assure results

Typologies of Result Based Monitoring Continuous Sustainability

Evaluation Criteria Relevancy Effectiveness Efficiencies Impact Sustainability

Impact Evaluation The objective of this type of evaluation is to determine whether or not a given project is achieving the intended impacts Impact project performance with training of project staff Impacts on income of people by apple farming project Impacts of irrigation on increment of income and reducing poverty

Impacts Assessment: Impacts vs. Counterfactual

Selection of Indicators of Project Outcomes Indicators are “the variables used to measure progress towards goals” (Stem , Margoluis, Salafsky, & Brown, 2005) Indicators are required to be selected for Monitoring and Evaluating Results

Indicators An indicators indicates progress, success, and failure of a policy, program and project Indicators are related: How do I know the intended positive change has taken place? What changes? In what way? (quantity, quality, time description of targets)

Types of Indicators I Input indicators Process indicators Output/outcome indicators Impact indicators Time related indicator Quantity related indicator Cost indicators Quality indicator

Steps to be Taken to Select Indicators Precisely identify what we want to monitor or examine List down the questions that need answers Identify the kind of indicators that can be used Review the methodological basis and robustness of these indicators Find out what data will be needed to derive each indicator, and where and how to get this data

SMART Indicator Examples of Indicators Specific Measurable Achievable Relevant Time-bound Examples of Indicators Drop out ratio Enrolment percentage

Targets and Baseline Target is the intended results Baseline is the existing scenario or reference point against which performance is compared Baseline is the existing scenario

M & E Plan The plan is managerial tool that: that specifies indicators, targets, sources of data, schedules, resources and responsibilities for M & E activities

Exercise

Methods of Evaluation Quantitative Qualitative Mixed

Quantitative Tools Questionnaire and survey Existing database

Qualitative Direct Observation Focus Group Discussion Interview Case Studies Document Studies Analysis of SWOT

Timing of Research When and How often to collect data necessary to provide required information

Data Analysis and Reporting Once data are collected, they are analyzed or processed for reporting. This may include averaging the data, calculating ratio, developing trends etc. Finally, the analysed data are reported in some form.

Essential Elements of Evaluation Report Executive Summary Main Body Outline of evaluation background (Objective, Strategy, Time/Duration, Role/ Introduction, Evaluator/s) Review of expenditure along with a narrative overview Evaluation log-frame and descriptive summary Results of each of the dimensions evaluated : Relevance, Effectiveness, Efficiency, Outcomes and impact and Sustainability Conclusion Summary of results, learning and recommendation Utility of results for management Justification of evaluation plan Changes in implementation modality at different points in time

Essential Elements of Evaluation Report Contd.. Annexes Initial outline used for implementation Evaluation log-frame and matrix demonstrating progress status Evaluation questions and results analyzed through the measurement of indicators Unprocessed data obtained from interview, survey, direct communication and other means Reference