Midterm Evaluation of GEOSS Highlights and Key Messages from the Evaluation Report Charles Hutchinson Chair, GEOSS Midterm Evaluation Team GEO Executive Committee Meeting Geneva, Switzerland XX July 2010

Introduction Evaluation Purpose The evaluation satisfies two needs: – Allows “midcourse” adjustments to GEOSS implementation – Establishes a reference point for future evaluations 2

Methodology A wide array of data sources informed the evaluation: Key Informant Interviews GEOSS User Survey GEO Work Plan Progress Reports Work Plan/Strategic Target alignment review Documentation and Literature review Case Studies of select Tasks 3 Terms Used- “key informants” - the individuals with active participation in GEO who participated in personal interviews “survey respondents” - individuals on the provider-user spectrum who provided input via web-accessible survey “stakeholders” - a combination of key informants, survey respondents and authors of the documents consulted during the evaluation.

Key Findings 1.GEOSS represents an important new Earth observation community and network. GEOSS has raised visibility of the importance and need for integrated global Earth observations. 2.Current GEOSS implementation sufficiently reflects high-level ministerial priorities including those contained in the Cape Town Declaration. 3.Stakeholders are generally positive about the foundation that has been established and optimistic that appropriate outcomes are being realized. 4

Key Findings 5.GEOSS implementation has created a path to enable full and open data sharing and lowered discussion barriers. 6.GEOSS implementation has resulted in positive outcomes for the Earth observation community, such as Data Sharing Principles. 5 4.GEOSS implementation has brought together various organizations and governments to collaborate and support integrated global Earth observations.

Key Findings 8.Survey respondents had an overall “neutral” feeling towards the status of GEOSS development and implementation at this point. 9.Stakeholders perceive that architecture developed by GEO does not yet meet their needs for data, information, and tools. 6 7.GEO has not adequately communicated evidence of progress to show value- added results unique to the implementation of GEOSS and to unequivocally prove a positive return on investment.

In Focus: Finding 9 Stakeholders perceive that architecture developed by GEO does not yet meet their needs for data, information, and tools. 7 Specific Issues: Confusion about Portal testing. QA/QC of material. Direct access to data and information rather than links to more portals or limited products.

Key Findings 10.Stakeholders are concerned about the sustainability of GEOSS with regard to (a) the voluntary nature of GEOSS implementation which has been beneficial up to this point for engaging partners; and, (b) the lack of sufficient resources, both financial and human to sustain efforts into the future. 11.Stakeholders indicated widely varying expectations for GEO and GEOSS, particularly as a source of new funding or a competing operational entity Some stakeholders view current GEO practices as co-opting achievements of contributors and giving them limited or no acknowledgement or credit.

In Focus: Finding 12 Some stakeholders view current GEO practices as co-opting achievements of contributors and giving them limited or no acknowledgement or credit. 9 Examples: GCOS / GEO Harmonization Engagement of UN Agencies GEONETCast Task ST Specific Issues: No standard for crediting material within the GCI (Task ST 09-02). Perceived reluctance to acknowledge existing frameworks which are major pieces of GEOSS. Desire by ALL parties to highlight their own role in activities. Perceived intent to create new GEO frameworks rather than working to support and connect existing, independent systems.

Key Findings 14.The GEOSS implementation approach does not explicitly describe an end-to-end process of how the application of resources supports the overall vision and goals of GEOSS, how or why benefits are expected, or when benefits will be achieved. Without this, it may be difficult for stakeholders to make well-informed decisions about supporting GEOSS GEO has not conducted a comprehensive gap analysis of either their implementation approach (structural) or observation needs (observational).

Recommendations Based on these key findings, the Evaluation Team makes the following 11

Recommendations 1.GEO should develop a long-term strategy to ensure the sustainability of GEOSS beyond GEO must investigate alternative models for sustained resource commitments from Members and Participating Organizations which are necessary for current and future operations GEOSS implementation in the short-term should be guided by an explicit approach linking activities and outputs of the GEO Work Plan to measureable, achievable objectives and strategic targets. This can be accomplished through adopting a logic model and performance measurement strategy.

In Focus: Recommendation 2 GEO must investigate alternative models for sustained resource commitments from Members and Participating Organizations which are necessary for current and future operations. 13 Specific considerations: Means to encourage “follow-through” on pledged support and contributions. Sustaining operations of GEO (i.e. Secretariat) at an effective level. Enabling GEO/GEOSS to support broad participation in GEOSS development (e.g. travel support for developing countries). Ensuring GEOSS will have a lasting presence in Earth observations. Elevating the status of GEOSS activities within contributing program offices.

Recommendations 5.GEO must improve its efforts in communication and outreach through: a)Clarifying their purpose to the stakeholder community; b)Enhancing clarity and traceability of GEO processes; c)Providing evidence of value-added results through GEOSS, and; d)Engaging a wider audience beyond those directly involved in GEOSS implementation GEO should clarify its role as a supporting and enabling platform by facilitating and providing value through coordination among existing Earth observation systems and developing an information networks system.

Recommendations 6.GEO should act to improve its understanding, engagement, and responsiveness to the user community by: a)undertaking a detailed characterization of its current users in order to strengthen and expand the user base; and, b)increasing opportunities for dialogue with the user community to provide helpful feedback on a timely basis. 15

Recommendations 7.GEO should conduct comprehensive observational and structural gap analyses as anticipated in the 10-Year Implementation Plan and Strategic Targets document. 8.GEO should establish clear and consistent mechanisms for properly attributing contributions to eliminate the appearance of co-opting activities. 16

Thank You! We will be happy to answer your questions at this time. If you have additional inquiries about the evaluation, please contact: [Who are we putting here? Is this necessary?] 17 Closing Notes: Electronic records from the evaluation, including planning documents, source material, and working papers have been transferred to the GEO Secretariat. The Evaluation Team also produced a supplementary report on Lessons Learned during the evaluation process with recommendations for future evaluation efforts.

Supplementary Slides 18

Supplementary Slide In Focus: Finding 8 Survey respondents had an overall “neutral” feeling towards the status of GEOSS development and implementation at this point 19 But, Key Informants had a much more positive outlook. Interpretations: The users are poorly-informed. The users have greater expectations for GEOSS. The users have not been engaged in GEOSS development and have little “buy-in.”

Supplementary Slide In Focus: Recommendation 5 GEO must improve its efforts in communication and outreach through: a)Clarifying their purpose to the stakeholder community; b)Enhancing clarity and traceability of GEO processes; c)Providing evidence of value- added results through GEOSS, and; d)Engaging a wider audience beyond those directly involved in GEOSS implementation. 20 Why “must”: Limited reach of the evaluation effort. Need to curb unrealistic expectations for GEOSS (e.g. as a funding mechanism). Need to show results to maintain support. Awareness should grow to understanding, understanding should grow to active engagement.

Supplementary Slide Evaluation Timeline January Evaluation Team established. – Jan. - Team receives guidance from M&E WG. February Development of detailed evaluation framework, data collection methods, and process schedule. March Data collection including interviews, web-based survey, and available literature – Mar. - Team ‘midway’ meeting in Geneva. April Data analysis, additional data collection, report drafting. May Complete draft sent for Secretariat factual review, development of findings and recommendations. – May - Report finalization meeting in Washington. June Report presented to M&E WG. 21

Supplementary Slide Lessons Learned The Evaluation requires dedicated staff support, in this case two full- time Team Members were key to completion. Evaluation Team should include multiple “types”- scientists and evaluators, but also policy makers, GEOSS users and contributors, and other experts as appropriate. Evaluation Teams would benefit from globally equitable composition. Some combination of dedicated project budget, extended timeline, and greater preparatory work by GEO would greatly increase the efficiency and effectiveness of the process. Additional time and resources are needed for outreach and engagement of the Earth observation stakeholder communities (particularly developing countries and marginal or potential users not already deeply involved in GEO). – A particular need is support for non-web-based outreach through print and in-person mechanisms. 22