GEOSS Monitoring and Evaluation Charles S. Baker (USA) M&E WG Co-Chair Joint meeting of M&E WG and ET January 20, 2010 Geneva, Switzerland 1

2 Requirement for GEOSS M&E 10-year Implementation Plan (2005): Item 7: Technical Approach, Capacity Building and Outreach: To monitor performance against the defined requirements and intended benefits. Item 5: Funding and Measuring Progress GEO will develop performance indicators for GEOSS. Resolution of the Third Earth Observation Summit (2005): Resolve to conduct a mid-term assessment of GEO by Cape Town Declaration (2007): We resolve to meet again before the end of 2010 to review progress, conduct a mid-term assessment and give further guidance on the implementation of GEOSS.

3 At GEO-V in Bucharest, Plenary agreed to Establish M&E WG Task ExCom with defining M&E WG TOR Task ExCom with moving process forward Open M&E WG to membership

4 Accomplishments to Date: M&E WG meetings First meeting in March 2009 Second meeting in July 2009 M&E Briefings to ExCom First briefing in June 2009 Second briefing in September 2009 M&E Briefing to GEO VI Plenary GEO-VI accepted M&E documents in Nov 2009 Nominations for Evaluation Team received

M&E Documents Produced 1.M&E WG Terms of Reference 2.M&E Framework Document 3.Summary Plan for the First Evaluation 5

6 Terms of Reference Define Purpose of M&E WG Define Objectives of M&E WG –Develop M&E Framework –Develop Evaluation Plan Define Approach and Functions of M&E WG –M&E process to be internal, non-complex and low cost –M&E to rely extensively on existing tools and instruments –Performance indicators to be uncomplicated, yet viable –M&E to be conducted by individuals designated by GEO members –M&E WG to provide oversight of the M&E process

Monitoring and Evaluation Framework Principles 1.M&E will be conducted within the perimeters of the GEO community; 2.M&E will take into account the non-binding nature of GEO; 3.M&E will make use of existing M&E tools and practices used in GEO; 4.M&E will address the full spectrum of GEOSS implementation activities; 5.M&E will use performance indicators for monitoring; 6.The M&E process will engage the user community; 7.M&E will provide feedback for improvement of future GEOSS planning and implementation; 8.Lessons learned from the conducted evaluation will inform improvements to the subsequent evaluation processes. 7

8 Principle 7. M&E will provide feedback for improvement of future GEOSS planning and implementation

9 Activitie s Outcomes Impacts What progress did we make towards achieving the targets? Evaluation: Did our activities produce intended outputs? Monitoring: Outputs Inputs GEOSS Logic Model

Approach to Monitoring Monitoring will track the progress of Task completion GEO Secretariat will perform monitoring Monitoring will be ongoing and systematic Monitoring will coincide with Work Plan reporting cycle Information will be collected through Task Sheets Task Sheet Template modified, based on M&E WG input, to: –Identify key milestones, with expected dates for their achievement –Identify quantitative indicators of progress, where appropriate –Enable collection of information on the use of portals –Separate planning from reporting information –Link overarching Tasks to Strategic Targets 10

Approach to Monitoring (contd) Work Plan Progress Report Secretariat will score Overarching Tasks (green, yellow, red) Progress Reports will show the score of the Sub-Tasks when necessary to explain the score of an Overarching Task; Secretariat will identify any gaps in the Work Plan (i.e., critical areas requiring work but for which no task is yet underway). Revised GEOSS Roadmap database will provide information for assessing progress toward achieving Targets; Secretariat will give an overall score for each Strategic Target; Distribution of Progress Report should be expanded to ensure that all parties involved in proposing, executing, or reviewing corrective actions are fully informed at all stages of the process. 11

Approach to Monitoring (contd) M&E WG will develop performance indicators for monitoring; Draft set of indicators will be presented to the ExCom for approval in late spring 2010; Upon approval of these indicators by ExCom, Secretariat will expand Progress Report to include a report on each of the indicators. 12

13 Approach to GEOSS Evaluation Establish Evaluation Team to conduct evaluation; New Evaluation Team will be appointed for each evaluation; First evaluation, mid-term assessment, based on Cape Town Declaration; Second evaluation to begin transition to regular cycle of evaluating GEOSS Targets: One GEOSS Transverse Area Target One GEOSS Societal Benefit Area Target Evaluation of unplanned contributing activities; Third and subsequent evaluations to be based on a regular cycle: Two or three Transverse Area Targets per year Three Societal Benefit Area Targets per year Future ministerial statements included in the scope of evaluations; Complete evaluation report(s) will be offered for factual review by the stakeholders.

14 GEOSS Building Blocks and SBAs

15 Summary Plan for the First Evaluation First evaluation will focus primarily on progress towards realizing priorities identified in the Cape Town Declaration; Other aspects of GEOSS implementation will be evaluated within the context of these priorities; Strategic Targets will be a key reference for clarifying the intended priorities and outcomes for GEOSS; Evaluation Team will make extensive use of Progress Reports and updated GEOSS Roadmap database; Draft evaluation report will be circulated within the GEO community for factual review; The Evaluation Team is solely responsible for the findings, conclusions and recommendations in the evaluation report. ExCom will coordinate response to the recommendations. 15

Cape Town Declaration We confirm our common view that: The sustained operation of terrestrial, oceanic, air-borne, and space-based observations networks is critical for informed decision making; Data interoperability is critical for the improvement and expansion of observational, modelling, data assimilation, and prediction capabilities; Continued research and development activities and coherent planning as essential for future observation systems; Continued cooperation and dialogue will establish GEOSS as a powerful means to support informed decision making; Coordination, at national, regional and global levels, continued investments, scientific and technological advances and innovative approaches to financing will be vital for upgrading and expanding Earth observations and building the capacity of individuals, institutions and systems, particularly in developing countries; We support the establishment of a process with the objective to reach a consensus on the implementation of the Data Sharing Principles for GEOSS to be presented to the next GEO Ministerial Summit. The success of GEOSS will depend on a commitment by all GEO partners to work together to ensure timely, global and open access to data and products; We commit to explore new ways and means for the sustained operations of the shared architectural GEOSS components and related information infrastructure; We commit ourselves to working together to improve the interoperability of and access to observation and associated prediction and information systems towards the continued strengthening of GEOSS and the full realization of the 10-Year Implementation Plan. 16

17 Evaluation Team M&E WG will appoint Chair/Co-Chairs. 17

18 Timeline of Evaluation Activities 18

BACK UP SLIDES 19

20 Guidelines for M&E WG Prepare final draft ToR Recognizing that M&E should be within GEO perimeter Recognizing that M&E should involve user community Recognizing that M&E should be independent of Committees Develop M&E Framework Observe the need for efficiency, simplicity and optimal use of resources Focus initially on monitoring, making use of available tools and instruments Develop a simple procedure for evaluation Recommend improvements to Task Sheets as the key monitoring tool Review and consolidate GEOSS Roadmap Develop parameters, criteria and indicators for evaluation

21 Guidelines for M&E WG contd. Develop Plan for First Evaluation Recognizing particular nature of GEO Recognizing the pilot nature of evaluation Recognizing the need to inform improvements in M&E framework Recognizing challenging schedule

Evaluation Questions What Should be Observed (Evaluation Criteria) Information Required To what extent do the Strategic Targets (and the associated outcomes) adequately reflect the priorities and commitments in the Cape Town Declaration? Strategic Targets include all significant aspects of the priorities and commitments in the Cape Town Declaration. Cape Town Declaration Strategic Targets document Evidence supporting the interpretation of the above documents. To what extent are the Work Plan Tasks reflective of the actions required to achieve the Strategic Targets? All Tasks are necessary to the achievement of the Strategic Targets. Processes are in place to identify gaps in the current Work Plan where additional Tasks are required to achieve Strategic Targets and to solicit offers to fill these gaps. Strategic Targets document GEOSS Roadmap Completed Task Sheets (current and past versions) Evidence regarding the relationship of Tasks to Targets. Evidence regarding work planning and gap analysis processes. To what extent are the Tasks being implemented as intended? Statement of targets and milestones in GEO planning documents in a form that permits objective verification against actual performance Actual performance reflects substantial achievement of performance expectations Completed Task Sheets (current and past versions). Work Plan Progress Reports and other information regarding the status of all Tasks. Information concerning the actual conduct of work planning, Task management, and reporting processes 22

To what extent are expected outcomes of GEOSS being realized? Clearly stated expected results that inform work planning and performance monitoring Achievement of near-term outcomes in some domains, although likely of limited scope Statement of expected outcomes for GEOSS in a form that depicts the logical relationships between GEOSS Tasks and the expected outcomes at various levels and/or time scales and between the expected outcomes and the ultimate intended impacts of GEOSS Evidence that statements of GEOSS outcomes influence the selection of Tasks Information regarding processes for the monitoring and assessment of outcomes related to GEOSS Evidence regarding the achievement of near-term GEOSS outcomes Are there significant unplanned benefits or unintended negative consequences of GEOSS? Documentation of unplanned benefits of GEOSS Absence of any significant negative unintended consequences. Information concerning the existence and magnitude of unintended consequences of participation in GEO, GEO Tasks or of GEOSS operations 23