Earth System CoG and the Earth System Grid Federation: A Partnership for Improved Data Management and Project Coordination NOAA ESRL Seminar April 8, 2014 Boulder, CO Sylvia Murphy (NOAA/CIRES) Luca Cinquini (JPL/NOAA), Cecelia DeLuca (NOAA/CIRES), Allyn Treshansky (NOAA/CIRES)

Presentation Outline ESGF-CoG Integration Overview of ESGF ESGF Architecture and Local Data Holdings Overview of CoG CoG Capabilities (Live Demo) ESGF-CoG Integration Development Tasks Upcoming Tutorials

ESGF-CoG Integration ESGF is an international, federated data archive focused on climate projects. CoG is a collaboration environment and hub to connect projects in the Earth sciences. CoG is going to become the new front-end for ESGF. – This will mean a superior interface to ESGF users and data managers in terms of: Overall usability Content management Model Intercomparison Project (MIP) support Multi-project support Online collaboration tools Reference: 3 rd Annual Earth System Grid Federation and Ultrascale Visualization Climate Data Analysis Tools Face-to-Face Meeting Report December (

ESGF Overview The Earth System Grid Federation (ESGF) is a multi-agency, international collaboration of people and institutions working together to build an open source software infrastructure for the management and analysis of Earth Science data on a global scale Collaboration led by PCMDI, includes institutions from several agencies from the U.S.A. (DOE, NASA, NOAA), Canada, Europe (IS-ENES-2), Australia and Asia ESGF manages and serves a global archive of climate data including: CMIP5 model output (basis of IPCC-AR5) Possibly the largest modeling effort in history: 40+ models, 25+ modeling centers, 17 countries, 2 PB of data Obs4MIPs: selected observations from NASA and DOE especially formatted for comparison and evaluation of CMIP5 models Ana4MIPs: reanalysis data also formatted as CMIP5 model output CORDEX: regional climate models, 2 PB of data TAMIP: atmospheric model intercomparison GeoMIP: geo-engineering model intercomparison DCMIP: atmospheric dynamical core model intercomparison WCRP recommended use of ESGF infrastructure for all future MIPs

ESGF System Architecture ESGF is a system of distributed and federated Nodes that interact dynamically through a Peer-To-Peer (P2P) protocol Distributed: data and metadata are published, stored and served from multiple centers (“Nodes”) Federated: Nodes interoperate because of the adoption of common services, protocols and APIs, and the establishment of mutual trust relationships Dynamic: Nodes can join/leave the federation dynamically – global data and services will change accordingly A client (browser or program) can start from any Node in the federation and discover, download and analyze data from multiple locations as if they were stored in a single central archive

ESGF Software Stack Software components can be grouped into 4 areas of functionality – the Node “flavors”: Data Node: secure data publication and access Index Node: metadata indexing and searching Identity Provider: user authentication and group membership Compute Node: analysis and visualization The ESGF software stack is based on the integration of several applications, APIs: – Open source engines (Postgres, Tomcat, Solr) – Geo-spatial servers (Thredds Data Server, Live Access Server) – Industry standards: OpenSSL, X509, OpenID, REST, … – Custom ESGF software Node flavors can be installed in various combinations depending on site needs, or to achieve higher performance and scalability All ESGF software is Open Source (BSD License) and freely available on GitHub –

ESGF ESRL Node NOAA/ESRL is hosting a full-featured ESGF Node: – Node system administrator: Doug Ohlhorst (big thanks!) Available data collections: – Ana4MIPs 20 th Century Reanalysis (Gil Compo, Cathy Smith) – DCMIP-2012 ( Atmospheric Dynamical Core Inter-Comparison workshop at NCAR, led by Christiane Jablonowski), including NOAA FIM model – QED-2013 (Quantitative Evaluation of Downscaling workshop at NCAR, sponsored by National Climate Projection and Prediction –NCPP- project) ESRL Node is part of ESGF federation: – ESRL collections can be accessed and discovered from other ESGF sites – Vice versa, a user can start from ESRL Node and find CMIP5 data throughout ESGF Vertical mesh layout from FIM test 5-1 (idealized tropical cyclone) conducted during DCMIP-2012.

Summary of ESGF Achievements ESGF represents a significant step forward for the management and access of climate data world-wide: Established the first global, distributed database of PB of climate model output and observations Data can be discovered through a federated faceted search or RESTful API Data download can be scripted and executed by programs Users need register only once, authenticate everywhere Architecture is scalable (for increased model and instrument resolution and rates) and extensible (to other formats, repositories and scientific domains) ESGF has established an open source collaboration across agencies and international boundaries Image courtesy of NCAR/CGD

Overview of CoG CoG is a collaboration environment and hub to connect projects in the Earth sciences. It hosts software development projects, model intercomparison projects (MIPS), university short-courses, and workshops. It includes a configurable search to data on ANY ESGF data node. It provides projects with a wiki and customizable navigation to wiki content. Projects, files, or pages can be made private. It contains an ontology for the description and management of projects and provides a consolidated look at this content across a project’s network. It contains a file server for documents and images. It provides services for Earth system model metadata collection and display. Some of the 74 projects hosted on CoG include: NOAA’s High Impact Weather Prediction Project (HIWPP) Atmospheric Dynamical Core Model Intercomparison Project (DCMIP) Reanalysis Data for CMIP5 (Ana4MIPs) Observational Data for CMIP5 (Obs4MIPs) National Unified Operational Prediction Capability (NUOPC) National Climate Predictions and Projections Platform (NCPP) Earth System Documentation (ES- DOC) Earth System Prediction Capability (ESPC) CoG Development Partners

Who’s Using CoG HIWPP (NOAA): NCPP (NOAA): Ana4MIPs (NOAA): NUOPC (Navy, USAF, NOAA): DCMIP-2012:

Wiki and Collaboration Tools The CoG layout is color- coded: The right-hand side (dark yellow) is where services (data, news, project connectivity) are located. The Upper Navigation bar (dark teal) contains links to project-level metadata. On the left (light teal) is an auto-generated navigation system created when projects develop freeform content. The central portion of the site is a wiki that allows projects to create their own content. Screenshot of the CoG project workspace for the 2012 Dynamical Core Model Intercomparison (DCMIP) Workshop.

Customizable Data Services…Interfacing with ESGF Search widget can be turned on/off. Search can be narrowed to any ESGF node and to any project (e.g. CMIP). Search facets can be created, deleted, and grouped. Help text can be added to the top of the search page. Search results can be saved to a Data Cart associated with a user. Items in the Data Cart persist. Search results can be: – Forwarded to the Live Access Server (LAS) for simple visualization. – Downloaded directly via a WGET script. – Associated with model metadata if it exists.

ESGF Search Customization downscaling-2013/

Data Cart Items in the Data Cart can be sent individually or collectively to LAS or WGET. The Data Cart is associated with a user and not a project.

Show Metadata

Project Networks and the Project Browser Projects in CoG are arranged in a hierarchy of Parents, Peers, and Children. The Project Browser displays the network and allows for inter-project navigation. Projects can be tagged with keywords and projects can be searched for using keywords.

CoG Schema The CoG schema contains classes to describe software development projects, short- courses or meetings, and overall project coordination. Projects select which metadata to display via a simple web form. Project-level metadata is linked in standardized locations via the upper navigation bar.

Project-level Metadata Roll-up Management of information is a major problem in projects that involve many sub-projects, partners, multiple leads, and many resources. CoG acts as an index into project information that is necessary for coordination and collaboration and enables people responsible for overall coordination to quickly get consolidated views of information. This example shows the Partners feature that allows projects to list their project partners and include a logo for each. Below the list for ED-DOC is a consolidated view of the partners for ES-DOC’s peer projects.

Resources Resources are pointers to data, files, and URLs. Resources folders can be created, moved, and deleted. Projects can turn on a set of standardized Resources folders (e.g. Presentations, Minutes). Saved data searches can be saved as a Resource. Each Resource can have a private wiki-based notes page to facilitate discussions.

News News is a way to send announcements across a project network. News is visible in the news widget on any targeted project. News will be added to social media (Google+, Facebook, Twitter, RSS) in a future release.

Model Metadata Services The CoG Team is partnering with the international Earth System Documentation (ES- DOC) project to develop and use an Earth System Model metadata entry and view capability. The ES-DOC Viewer is a lightweight JavaScript plugin that will display any Common Information Model (CIM) record. The ES-Questionnaire collects standardized CIM metadata through a high-customizable web form. The output is saved to a community CIM repository.

CoG-ESGF Future Work Requirements are coming from HIWPP, CMIP6, the ESGF integration, and other projects. CMIP6 will include a set of interconnected MIPs. – Work is starting on the CMIP6 sites. CoG is going to replace the ESGF web front end. – Work should be completed by the end of the summer 2014 with a production system in place by the end of the year. – CoG will be federated so that projects hosted on one CoG-ESGF instance will be visible on others. CoG is being modified to conform to Federal and DOC requirements. OpenID access will be added to CoG, which will improve the security of the site. The local CoG URL will be changed to a.gov address.

Webinar/Tutorials Fridays at 10am Mountain Time Contact for more Other group or individual sessions available on demand. Scheduled Sessions: 11 Apr: HIWPP 02 May: ESPC

Questions? CoG: ESRL ESGF data node: PCMDI ESGF data node: JPL ESGF data node: