1. © GEO Secretariat 2.4 Monitoring and Evaluation Activities Craig Larlee / John Adamec Co-Chairs, M&E Working Group GEO 2015 Work Plan Symposium 5—7 May 2015 Geneva, Switzerland

2. Outline ● Overview of Monitoring and Evaluation activities during GEO's first decade. ● Summary of findings and recommendations from the first five evaluations. ● Progress in implementing evaluation recommendations. ● Interim findings of the sixth (summative) evaluation. ● Lessons for the second decade.

3. Overview of M&E Activities ● 10 year Implementation Plan: “GEO will develop performance indicators for GEOSS”. ● 2006: Several Members at GEO-III express interest in developing indicators of GEOSS outcomes. ● 2007: Montreal workshop; Report to GEO-IV. ● 2008: Paris workshop; GEO-V creates M&E Working Group. Strategic Targets document developed. ● 2009: First meetings of the M&E WG; Launch of the first (Mid-Term) evaluation; Approval by GEO-VI of M&E Framework Document. ● 2010: M&E WG begins looking at performance indicators; Launch of 2 nd evaluation (Architecture, Data Mgmt); Mid-Term Evaluation reported to GEO-VI. Evaluation schedule revised.

4. Overview of M&E Activities ● 2011: Test set of 18 indicators developed for transverse areas. Launch of 3 rd evaluation (Agriculture, Biodiversity, Ecosystems); 2 nd evaluation reported to GEO-VII. Change to the new Work Plan Management approach. ● 2012: Launch of 4 th evaluation (Disasters, Energy, Health); 3 rd evaluation reported to GEO-VIII; ExCom requests assessment of implementation of evaluation recommendations. Boards assess progress toward Strategic Targets. ● 2013: Launch of 5th evaluation (Weather, Water, Climate); 4th evaluation reported to GEO-IX. First assessment of implementation of evaluation recommendations. ● 2014: Launch of 6 th evaluation (summative); 5 th evaluation reported to GEO-X.

5. Evaluation Findings (Common Themes E 1-4) ● Lack of clear objectives against which progress may be evaluated – Strategic Targets too broad, not achievable by 2015. – Work planning system (“component sheets”) incomplete and inefficient. ● Gap between objectives and actions taken not managed systematically – Process needed to systematically identify and prioritize areas where additional resources are most critical. ● User perspective often not clearly visible – Difficult to obtain user responses during evaluations: contact lists out of date, users contacted often unfamiliar with GEO, no taxonomy of users.

6. Evaluation Findings (Common Themes E 1-4) ● Resource constraints limited achievement – Contributions did not always materialize. – Lack of an overall strategy to identify requirements and seek targeted contributions. ● Weaknesses in management processes – Information not always flowing back from tasks. – Collaboration across SBAs was challenging. – Global initiatives necessitated different kind of management. ● Communications, Role and Reporting – Difficulties in articulating: GEO role and added value, progress and achievements.

7. Implementation of Recommendations ● Most recent assessment is the one presented to GEO-XI (below). ● Action has been taken on most recommendations, but many are being addressed through the IPWG process. ● Completed ≠ Fully implemented. ● Deferred recommendation deals with performance indicators.

8. Interim Findings of 6 th Evaluation 1.GEO provides a common flexible framework for international collaboration in Earth observations. 2.Implementation of GEOSS open data sharing principles is a principal accomplishment of GEO. 3.Global initiatives are successful, but GEO has not taken full advantage of key international initiatives for mutual benefit. 4.Insufficient implementation of the GEO Work Plan has led to only fair achievement of Strategic Targets. 5.The lack of in-situ Earth observations remains a significant gap in GEOSS.

9. Interim Findings of 6 th Evaluation 6.GEO must significantly strengthen user engagement and relevance to decision-making to achieve success. 7.GEO’s progress in capacity building is uncertain, but has established clear channels for considerable improvement in the future, especially in developing countries. 8.The GEOSS Common Infrastructure (GCI) has greatly advanced interoperability, despite the GEOSS Portal failing to meet user needs. 9.There is a recurrent lack of resources for the implementation of GEOSS and an uneven commitment from GEO members.

10. Lessons for GEO's Second Decade ● There have been many successes: GCI, data sharing principles, global initiatives, visible progress in most tasks. ● Weaknesses are also clear – same issues have come up in all evaluations. – Clarity/measurability of objectives; Understanding user needs; Identifying gaps and mobilizing resources; Following through on commitments; Demonstrating value to others. ● IPWG Strategic Plan is clearly built on awareness of these issues. ● But...need to keep them in mind as plans are implemented and new structures and processes are defined.