GEF Expanded Constituency Workshop Kigali, rwanda February 2019 Sara El Choufi - Evaluation Analyst Photo by Karsten Würth (@inf1783) on Unsplash

Overview Monitoring and Evaluation in the GEF Independent Evaluation Office (IEO) M&E Policy Terminal Evaluation Guidelines Recent Evaluation Findings Ongoing Evaluations

Is our activity on track? Monitoring Is our activity on track? Monitoring uses systematic collection of data to keep activities on track. Forms of monitoring: Monitoring of environmental conditions and stressors Monitoring of progress toward project/program outcomes Monitoring of project/program performance

Evaluation Are we doing the right thing? Are we doing things right and efficiently? Are there better ways of doing it? Evaluation is a systematic assessment of an activity (program, strategy, etc.) that assesses relevance, effectiveness, efficiency, results and sustainability. Project/Program Evaluations: Mid-term Terminal Other forms of evaluation: impact, thematic, performance, country, corporate, comprehensive

M&E in the GEF Two overarching objectives: Promote accountability for the achievement of GEF objectives through the assessment of results, effectiveness, processes, and performance of the partners involved in GEF activities. Promote learning, feedback, and knowledge sharing on results and lessons learned among the GEF and its partners as a basis for decision making on policies, strategies, program management, programs, and projects; and to improve knowledge and performance.

Separate reporting lines for Monitoring (through Secretariat) and Evaluation (through IEO) GEF Council Annual evaluation reports Overall Performance Study (to Assembly) Annual Work Program and Budget Annual Monitoring Report Evaluation Management Response Programming documents and indicators Results Based Management Agency evaluation units GEF Independent Evaluation Office GEF Secretariat Project and Program Implementation Reports Agency Portfolio Reports Project documents with M&E plans Project and Program evaluations Corporate evaluations Project and Program Independent evaluations Agency GEF coordination units Project and Program Implementation Reports Project and Program monitoring documentation Terminal evaluations GEF projects and programs

GEF Independent Evaluation Office Mission: Enhance global environmental benefits through excellence, independence, and partnership in monitoring and evaluation. Functions: Accountability Promote knowledge sharing and Learning Brief history: 1996 — Initially established as an M&E unit within the GEF Secretariat 2003 — The M&E unit was made autonomous of the GEF Secretariat 2005 — The unit was renamed as GEF Evaluation Office 2013 — The office was renamed as GEF Independent Evaluation Office GEF IEO Functions: Independent Evaluation: The main function of the IEO is to independently evaluate the GEF’s efficiency, effectiveness, relevance, impact, and sustainability. Normative function: IEO sets minimum monitoring and evaluation standards within the GEF to ensure improved and consistent measurement of the GEF results . Oversight function: IEO provides quality control for the minimum requirements of monitoring and evaluation practices in the GEF, in full cooperation with relevant units in the GEF Agencies, and tracks implementation of Council decisions related to evaluation recommendations. Knowledge management and dissemination function: IEO supports knowledge sharing and follow-up of evaluation recommendations. It participates in the development and maintenance of a comprehensive knowledge system based on evaluation findings and lessons.

GEF IEO Stakeholders Clients with governance role: GEF Council GEF Assembly The replenishment group Clients that carry out decisions of the governing bodies GEF Secretariat GEF Agencies Executing agencies at the country or regional level Country Clients Clients involved in monitoring and evaluation Wider audience: environmental entities, academia, research institutions, civil society, general public

The GEF M&E Policy Defines the concepts, role, and use of monitoring and evaluation within the GEF Defines the institutional framework and responsibilities Indicates the GEF minimum M&E requirements covering: project design application of M&E at the project level project evaluation engagement of Operational Focal Points in M&E

The GEF M&E Policy: 4 Minimum Requirements 1. Design of M&E Plan Concrete and fully budgeted M&E plan Align with GEF focal area results frameworks Baseline data Mid-term reviews and Terminal Evaluations Organizational set up and budget 2. Application of M&E Plans SMART indicators Baseline for the project is fully established Organizational set up for M&E is operational and its budget is spent as planned 3. Project and Program Evaluation All FSPs will be evaluated and evaluations are Independent Apply evaluation norms and standards Assess, outputs and outcomes, likelihood of sustainability 4. Engagement of Operational Focal Points OFPs should be informed on M&E activities GEF Agencies keep track of the application of this requirement

Forthcoming Updates to the M&E Policy Separation of M&E policies consistent with GEF’s evolution Expanded coverage of GEF Priorities based on IEO Gap Analysis (2015) gender, safeguard standards, stakeholder engagement, other Focus on programmatic approaches requires attention to program-level M&E (OPS6) Evolution: GEF formally introduced M&E in the TORs in 2003, highlighting the independent nature of these functions. The M&E Policy followed in 2006 and was updated in 2010, to reflect the evolution of the GEF as an institution. In both occasions IEO and SEC worked jointly at the update. IIEO Gap Analysis: in its AWPB for GEF-6 (2014), IEO indicated it will update the M&E policy. One year later, IEO provided an analysis of current gaps in the 2010 policy. The analysis identified several gaps… OPS6: the GEF is different from what it was 8 years ago. It increasingly invests in integrated programming through programs and multifocal projects. Most of the time, program-level M&E is not addressed, and when it is, it does not report on how child projects contribute to the program’s objectives. Current reporting requirements for multifocal projects increase operating costs; at the same time, synergies generated and tradeoffs mitigated are not captured

Underpinning IEO Work: Terminal Evaluations Terminal Evaluations Include: Results: Outputs, outcomes and progress to impact Implementation, execution, and project cycle related information Project finances including co-financing Recommendations and Lessons for the future GEF M&E Policy (2010): Minimum Requirement 3 Terminal evaluations mandatory since 1995 Required for full-sized and medium-sized projects Terminal Evaluation Guidelines: Available at https://www.gefieo.org/ev aluations/guidelines-gef- agencies-conducting- terminal-evaluation-full- sized-projects What are terminal Evaluations? Terminal evaluations are evaluations at the end of the project. How are TEs used? They are an important sources of information on a project: please see the slide TEs are also used for: Reporting at the project portfolio level (APR, AMR) Input to other evaluations STAR’s performance index More than a 1300 terminal evaluations are completed so far. Terminal evaluation may be accessed at: GEF website: http://www.thegef.org/projects Through PMIS

Characteristics of a good terminal evaluation Quality Outcomes Consistency and comprehensiveness Sustainability Lessons and recommendations Project finances M&E Other characteristics Transparency and timeliness Candor Balance Utility Characteristics of a good terminal evaluation Quality of terminal evaluations   The GEF IEO, along with the evaluation offices of some of the GEF agencies, track quality of terminal evaluations. The GEF IEO approach assesses quality on following parameters: Outcomes: how well the outcomes and other results of the project have been discussed and compared with the expectations. Consistency and comprehensiveness: to what extent information provided in the terminal evaluation is consistent, comprehensive and complete. Sustainability: how well the terminal evaluation report discusses concerns related to future progress and continuation of the progress made so far. Lessons and recommendations: the extent to which a terminal evaluation provides valid recommendations and lessons that are based on the project experience. Project finances: the extent to which information on project finances including information on co-financing is complete. M&E: the extent to which the terminal evaluation provides information on implementation of the M&E plan, and use of monitoring information. Among these criteria, the GEF IEO gives more weightage to discussion on outcomes, and consistency and comprehensiveness of information. There are several other characteristics of the good terminal evaluation that are not covered in the GEF IEO criteria. These may also play an important role in preparation of terminal evaluation of good quality: Transparency: the extent to which the process through which terminal evaluation was conducted adds to its credibility. Timeliness: timeliness of the terminal evaluation ensures that the evaluators had access to relevant information sources and it ensures that the information coming out of the evaluation is readily available to the GEF stakeholders. Candor: Candor is very important in ensuring that the weaknesses identified in the project results, implementation process, and design, are documented, analyzed and presented so that these may be avoided in future. Balance: the evaluation should provide information on different perspectives on project performance. It should provide a balanced coverage to these perspectives based on their relevance. Utility: a good terminal evaluation should also be backed up with efforts to ensure its utility. Thus, it should be part of a larger knowledge management system that ensures it utility to the partnership.

Independent Evaluations and Methods Semi-Annual Evaluation Report Impact, Thematic, Performance, Corporate, Country Clusters Comprehensive Evaluations every 4 years Methods Qualitative Quantitative including GIS, Remote Sensing, Big Data Analytics

RECENT EVALUATION FINDINGS

Performance and Impact 81% 62% Satisfactory outcomes of projects have outcomes that are likely to be sustained Outcomes: Performance and sustainability of outcomes is higher in MICs ((Brazil, China, India, Mexico, Russian Federation) A lower percentage of projects implemented in Africa, Small Island Developing States (SIDS), and Least Developed Countries (LDCs) have satisfactory outcomes as compared with the portfolio average. Africa – 74% SIDS – 64% LDCs – 71% When projects are divided into two categories – those approved from GEF-pilot trough GEF-2 and those from GEF-3 and later, projects implemented in fragile states, regional projects, and in Europe and Central Asia show statistically significant improvement over time. In these groups we see improvement of 10% or more. Fragile states – 58% (upto GEF-2); 77% (GEF-3 onwards) Regional projects – 76% (upto GEF-2); 85% (GEF-3 onwards) Europe and Central Asia – 76% (upto GEF-2); 86% (GEF-3 onwards) Sustainability: 62 percent of completed GEF projects are rated likely for outcome sustainability. Sustainability have improved over the GEF replenishment periods. Some categories of projects such as projects in LDCs, fragile states, global projects, climate change and biodiversity focal area, show statistically significant improvements in sustainability ratings through GEF-2 to GEF-3 and onwards. Likelihood of outcome sustainability at project completion is influenced by the quality of project preparation, country context, support from the government, the quality of implementation and execution, and materialization of co-financing. Cofinancing commitments for GEF-6 exceed the target Project cycle efficiency gains are slow 8.8:1 6:1 target

Broader adoption and transformational change 61% 59% of projects achieved broader adoption of projects achieved environmental stress reduction GEF interventions have contributed to reducing environmental stress and broader adoption ✔ Level of ambition ✔ Effective transformational mechanism ✔ Quality of implementation and execution ✔ Harnessing market forces ✔ May be achieved by projects of different size Mechanisms for broader adoption Mainstreaming and replication Scaling-up and market change Success factors for transformational change: Clear ambition in designs Addressing market reforms through policies Mechanisms for financial sustainability Quality of implementation and execution May be achieved by projects of different size

GEF interventions yield positive returns on investment $1:1.08 $1:1.04 Land degradation Biodiversity 43.52 tC/ha Picture of land degradation Picture of biodiversity $ amount on top of the picture Land Degradation (take from old ppt) $1.08/$1 Value for Money of Biodiversity Projects $1.04 return per $1 invested (likely underestimate) Positive impacts on forest cover, vegetation productivity, and carbon sequestration Factors (BD) Project Size

GEF’s Comparative advantage: OPS6 RELEVANCE Serves multiple conventions and broad range of environmental issues Strong Support to LDCs and SIDS PERFORMANCE Long history of good performance Ability to address linkages and synergies between focal areas TRANSFORMATIONAL Ability to create an enabling environment through legal and regulatory reforms Delivers innovative financial models and risk-sharing approaches Picture of land degradation Picture of biodiversity $ amount on top of the picture Land Degradation (take from old ppt) $1.08/$1 Value for Money of Biodiversity Projects $1.04 return per $1 invested (likely underestimate) Positive impacts on forest cover, vegetation productivity, and carbon sequestration Factors (BD) Project Size

Sharing Evaluative Knowledge Events and Products Website Earth-Eval Partnerships Expanded Constituency Workshops

EVALUATION WORK IN PROGRESS Photo by Jeremy Cai on Unsplash

EVALUATION WORK IN PROGRESS Evaluations Underway June 2019 Annual Performance Report with a focus on Transportation Evaluation of GEF Support to Scaling Up Impacts Value for Money Analysis of GEF Support to Sustainable Forest Management and REDD+ projects The GEF Evaluation Policy Fall 2019 Strategic Country Cluster Evaluations: African Biomes LDCs SIDS

SUSTAINABILITY Photo by David Clode on Unsplash

Sustainability at Project Completion 62% of projects have outcomes that are likely to be sustained GEF’s definition of sustainability is consistent with other multilateral organizations: continuation of benefits after the [project] implementation and risks to their continuation. At project completion, 62% of projects are likely to be sustained. Outcomes: Performance and sustainability of outcomes is higher in MICs ((Brazil, China, India, Mexico, Russian Federation) A lower percentage of projects implemented in Africa, Small Island Developing States (SIDS), and Least Developed Countries (LDCs) have satisfactory outcomes as compared with the portfolio average. Africa – 74% SIDS – 64% LDCs – 71% When projects are divided into two categories – those approved from GEF-pilot trough GEF-2 and those from GEF-3 and later, projects implemented in fragile states, regional projects, and in Europe and Central Asia show statistically significant improvement over time. In these groups we see improvement of 10% or more. Fragile states – 58% (upto GEF-2); 77% (GEF-3 onwards) Regional projects – 76% (upto GEF-2); 85% (GEF-3 onwards) Europe and Central Asia – 76% (upto GEF-2); 86% (GEF-3 onwards) Sustainability: 62 percent of completed GEF projects are rated likely for outcome sustainability. Sustainability have improved over the GEF replenishment periods. Some categories of projects such as projects in LDCs, fragile states, global projects, climate change and biodiversity focal area, show statistically significant improvements in sustainability ratings through GEF-2 to GEF-3 and onwards. Likelihood of outcome sustainability at project completion is influenced by the quality of project preparation, country context, support from the government, the quality of implementation and execution, and materialization of co-financing.

Discussion Based on your experience, discuss What are the factors that affect sustainability of project outcomes? Sustainability: 62 percent of completed GEF projects are rated likely for outcome sustainability. Sustainability have improved over the GEF replenishment periods. Some categories of projects such as projects in LDCs, fragile states, global projects, climate change and biodiversity focal area, show statistically significant improvements in sustainability ratings through GEF-2 to GEF-3 and onwards. Ask participants to discuss their experience with GEF and other environmental projects. What are the factors that affect sustainability? Give participants 10 minutes for small group discussion. Ask several groups to share their reflections with the whole group. Make notes on the flipchart. Then, discuss the IEO findings: GEF IEO sustainability analysis (June 2018, part of the Annual performance Report). The following factors affect the likelihood of sustainability of outcomes at project completion: the quality of project preparation, country context, support from the government, the quality of implementation and execution, and materialization of co-financing.

Discussion - continued What factors affect sustainability after the project completion? We discussed some of the factors that affect project sustainability before and during the project, such as quality of project design, project implementation and execution, government support, and country context. Now, based on your experience, what are some of the factors that may affect sustainability of outcomes once the project is completed? Discussion in small groups – 5 minutes, followed by a plenary discussion. Make notes on the flipchart, then discuss the IEO findings (GEF IEO sustainability analysis (June 2018, part of the Annual performance Report): 53 projects were evaluated in-depth for sustainability after the project completion. The key factors that contribute to higher sustainability of outcomes after the project completion include high stakeholder buy-in, political support, availability of financial support for follow up, and sustained efforts from the executing agency.

VISIT US AT www.gefieo.org FOR MORE VISIT US AT www.gefieo.org Photo by Karsten Würth (@inf1783) on Unsplash