Development of an Integrated Earth System Climate Modeling Environment

Slides:

Advertisements

Similar presentations

DATE: 2008/03/11 NCHC-Grid Computing Portal (NCHC-GCE Portal) Project Manager: Dr. Weicheng Huang Developed Team: Chien-Lin Eric Huang Chien-Heng Gary.

Advertisements

Building Portals to access Grid Middleware National Technical University of Athens Konstantinos Dolkas, On behalf of Andreas Menychtas.

LEAD Portal: a TeraGrid Gateway and Application Service Architecture Marcus Christie and Suresh Marru Indiana University LEAD Project (

TeraGrid Deployment Test of Grid Software JP Navarro TeraGrid Software Integration University of Chicago OGF 21 October 19, 2007.

ESA Data Integration Application Open Grid Services for Earth Observation Luigi Fusco, Pedro Gonçalves.

A Workflow Engine with Multi-Level Parallelism Supports Qifeng Huang and Yan Huang School of Computer Science Cardiff University

Earth System Curator Spanning the Gap Between Models and Datasets.

Metadata Development in the Earth System Curator Spanning the Gap Between Models and Datasets Rocky Dunlap, Georgia Tech.

1 G2 and ActiveSheets Paul Roe QUT Yes Australia!

RCAC Research Computing Presents: DiaGird Overview Tuesday, September 24, 2013.

M. Lautenschlager, H. Ramthun 1 Metafor Review 5 / 2010.

© , Michael Aivazis DANSE Software Issues Michael Aivazis California Institute of Technology DANSE Software Workshop September 3-8, 2003.

Peoplesoft: Building and Consuming Web Services

System Design/Implementation and Support for Build 2 PDS Management Council Face-to-Face Mountain View, CA Nov 30 - Dec 1, 2011 Sean Hardman.

A Quick Tour of the NOAA Environmental Software Infrastructure and Interoperability Group Cecelia DeLuca and the ESMF team ESRL Directorate Seminar June.

Sergey Belov, Tatiana Goloskokova, Vladimir Korenkov, Nikolay Kutovskiy, Danila Oleynik, Artem Petrosyan, Roman Semenov, Alexander Uzhinskiy LIT JINR The.

Apache Airavata GSOC Knowledge and Expertise Computational Resources Scientific Instruments Algorithms and Models Archived Data and Metadata Advanced.

Z EGU Integration of external metadata into the Earth System Grid Federation (ESGF) K. Berger 1, G. Levavasseur 2, M. Stockhause 1, and M. Lautenschlager.

CCSM Portal/ESG/ESGC Integration (a PY5 GIG project) Lan Zhao, Carol X. Song Rosen Center for Advanced Computing Purdue University With contributions by:

OnTimeMeasure Integration with Gush Prasad Calyam, Ph.D. (PI) Tony Zhu (Software Programmer) Alex Berryman (REU Student) GEC10 Selected.

Metadata Creation with the Earth System Modeling Framework Ryan O’Kuinghttons – NESII/CIRES/NOAA Kathy Saint – NESII/CSG July 22, 2014.

The Earth System Curator Metadata Representations Prototype Portal in Collaboration with ESMF and ESG Rocky Dunlap Spencer Rugaber Georgia Tech.

GDT V5 Web Services. GDT V5 Web Services Doug Evans and Detlef Lexut GDT 2008 International User Conference August 10 – 13  Lake Las Vegas, Nevada GDT.

Presented by The Earth System Grid: Turning Climate Datasets into Community Resources David E. Bernholdt, ORNL on behalf of the Earth System Grid team.

Enabling computational modeling and geospatial data analysis through HUBzero Venkatesh Merwade, Lan Zhao, Carol Song Purdue University Hubbub2013, September.

Towards a Javascript CoG Kit Gregor von Laszewski Fugang Wang Marlon Pierce Gerald Guo

A Metadata Catalog Service for Data Intensive Applications Presented by Chin-Yi Tsai.

Metadata for the Coupled Ocean/Atmosphere Mesoscale Prediction System (COAMPS) using the Earth System Modeling Framework (ESMF) Peter Bosler University.

NE II NOAA Environmental Software Infrastructure and Interoperability Program Cecelia DeLuca Sylvia Murphy V. Balaji GO-ESSP August 13, 2009 Germany NE.

OpenSees on NEEShub Frank McKenna UC Berkeley. Bell’s Law Bell's Law of Computer Class formation was discovered about It states that technology.

ESP workshop, Sept 2003 the Earth System Grid data portal presented by Luca Cinquini (NCAR/SCD/VETS) Acknowledgments: ESG.

INFSO-RI Enabling Grids for E-sciencE Logging and Bookkeeping and Job Provenance Services Ludek Matyska (CESNET) on behalf of the.

ATMOSPHERIC SCIENCE DATA CENTER ‘Best’ Practices for Aggregating Subset Results from Archived Datasets Walter E. Baskin 1, Jennifer Perez 2 (1) Science.

The PROGRESS Grid Service Provider Maciej Bogdański Portals & Portlets 2003 Edinburgh, July 14th-17th.

Integrated Model Data Management S.Hankin ESMF July ‘04 Integrated data management in the ESMF (ESME) Steve Hankin (NOAA/PMEL & IOOS/DMAC) ESMF Team meeting.

The Arctic Observing Network (AON) Cooperative Arctic Data and Information Service (CADIS) Florence Fetterer,

GEM Portal and SERVOGrid for Earthquake Science PTLIU Laboratory for Community Grids Geoffrey Fox, Marlon Pierce Computer Science, Informatics, Physics.

Workflow Architecture Create CaseConfigure CaseSubmit Case Authentication/ Authorization Track Status Post-processDebugging Transfer Files Publish Metadata.

TeraGrid Advanced Scheduling Tools Warren Smith Texas Advanced Computing Center wsmith at tacc.utexas.edu.

1 Earth System Modeling Framework Documenting and comparing models using Earth System Curator Sylvia Murphy: Julien Chastang:

SAN DIEGO SUPERCOMPUTER CENTER Inca TeraGrid Status Kate Ericson November 2, 2006.

Evolving toward a Coherent, Collaborative Framework for Earth Science Data, Tools and Services Christopher Lynnes, Kwo-Sen Kuo and Kevin Murphy Earth Science.

A Data Access Framework for ESMF Model Outputs Roland Schweitzer Steve Hankin Jonathan Callahan Kevin O’Brien Ansley Manke.

A Practical Approach to Metadata Management Mark Jessop Prof. Jim Austin University of York.

May 6, 2002Earth System Grid - Williams The Earth System Grid Presented by Dean N. Williams PI’s: Ian Foster (ANL); Don Middleton (NCAR); and Dean Williams.

Strategic Plan Implementation Cecelia DeLuca/NCAR (ESMF) December 17, 2008 ESMF Board/Interagency Meeting.

Exporting WaterML from the Earth System Modeling Framework Xinqi Wang Louisiana State University NCAR SIParCS Program August 4, 2009.

1 Adventures in Web Services for Large Geophysical Datasets Joe Sirott PMEL/NOAA.

1 Accomplishments. 2 Overview of Accomplishments  Sustaining the Production Earth System Grid Serving the current needs of the climate modeling community.

Towards Self-Describing Workflows for Climate Models Kathy Saint – UCAR Ufuk Utku Turuncoglu – ITU Sylvia Murphy – NCAR Cecelia DeLuca – NCAR.

Earth System Curator and Model Metadata Discovery and Display for CMIP5 Sylvia Murphy and Cecelia Deluca (NOAA/CIRES) Hannah Wilcox (NCAR/CISL) Metafor.

1 Summary. 2 ESG-CET Purpose and Objectives Purpose  Provide climate researchers worldwide with access to data, information, models, analysis tools,

Building Community and Capability through Common Infrastructure: ESMF and the Earth System Curator Cecelia DeLuca MAP Meeting College.

AHM04: Sep 2004 Nottingham CCLRC e-Science Centre eMinerals: Environment from the Molecular Level Managing simulation data Lisa Blanshard e- Science Data.

ESMF and the future of end-to-end modeling Sylvia Murphy National Center for Atmospheric Research

Application of RDF-OWL in the ESG Ontology Sylvia Murphy: Julien Chastang: Luca Cinquini:

Application Web Service Toolkit Allow users to quickly add new applications GGF5 Edinburgh Geoffrey Fox, Marlon Pierce, Ozgur Balsoy Indiana University.

Climate-SDM (1) Climate analysis use case –Described by: Marcia Branstetter Use case description –Data obtained from ESG –Using a sequence steps in analysis,

Climate Modeling in a Browser The Development of an Earth System Climate Modeling Science Gateway Carol X. Song, Ph.D. Senior Research Scientist Rosen.

Metadata Development in the Earth System Curator Spanning the Gap Between Models and Datasets Rocky Dunlap, Georgia Tech 5 th GO-ESSP Community Meeting.

Origami: Scientific Distributed Workflow in McIDAS-V Maciek Smuga-Otto, Bruce Flynn (also Bob Knuteson, Ray Garcia) SSEC.

Simulation Production System Science Advisory Committee Meeting UW-Madison March 1 st -2 nd 2007 Juan Carlos Díaz Vélez.

The Virtual Observatory and Ecological Informatics System (VOEIS): Using RESTful architecture and an extensible data model to provide a unique data management.

REST API to develop application for mobile devices Mario Torrisi Dipartimento di Fisica e Astronomia – Università degli Studi.

The Earth System Curator Metadata Infrastructure for Climate Modeling Rocky Dunlap Georgia Tech.

Metadata Support for Model Intercomparison Projects Sylvia Murphy: Cecelia DeLuca: Julien.

A Quick Tour of the NOAA Environmental Software Infrastructure and Interoperability Group Cecelia DeLuca Dr. Robert Detrick visit March 28, 2012

Data Bridge Solving diverse data access in scientific applications

Metadata Development in the Earth System Curator

Data Management Components for a Research Data Archive

Presentation transcript:

Development of an Integrated Earth System Climate Modeling Environment Purdue University/RCAC NCAR/ESG NOAA/CIRES

A Great Team! Purdue/RCAC Carol X. Song, Lan Zhao, Chris Thompson, Robert Campbell, Jungha Woo, Han Zhang, Madhavan Lakshminarayanan NCAR Don Middleton, Nathan Wilhelmi, Eric Nienhouse, Michael Burek, Rick Brownrigg NOAA/CIRES Kathy Saint (SGI), Cecelia DeLuca, Sylvia Murphy “Developing an integrated end-to-end TeraGrid climate modeling environment”, TeraGrid 2011 conference, Salt Lake City, UT, July 2011. 05/11/2012 XSEDE Science Gateway

Background Motivation for a climate modeling environment Project goals CCSM/CESM Resource Expertise A collaborative online environment Existing synergistic community efforts Earth System Grid, Earth System Curator, ESMF, METAFOR Common Information Model and Purdue CCSM3 portal Project goals Provide an integrated environment for climate modeling, data/metadata archival, visualization and analysis Support research and education, lower barriers to using climate models 05/11/2012 XSEDE Science Gateway

Authentication/ Authorization System Architecture CESM portal ESG gateway Account DB User requests TG MyProxy Token Mgr CESM Web Services SOAP and Restful XML/JSON Create Case Configure Case Submit Case Track Status Data/ Metadata Authentication/ Authorization Transfer Files Job Management Debugging Post-process Publish Data Publish Metadata XD community account/Individual account Output Output Jobs ESG Data Publisher Scratch Storage iRODS 05/11/2012 XSEDE Science Gateway

CESM Portal - Model Workflow Insulate users from cluster Hide command line details Integrate CESM 1.0 + metadata capabilities + ESG publishing + NCL Web services (SOAP, Restful XML/JSON) Manage user’s experience Maintain job status and metrics End-to-end modeling from simulation to data/metadata publishing Bind independent processes into unified workflow CESM Portal JBOSS Job Scripts Publish Scripts SOAP CESM-WS/ gateway Create case CESM Model Create create_case Configure case Configure ./configure Compile <case>.compile Compile case Execute <case>.submit Query Status Execute case Data Configure post-process Post-processing Post-processing Engines Run post-process AMWG NCL scripts ESG Publication workflow … 05/11/2012 XSEDE Science Gateway

ESMF Attributes and Self-describing Models Metadata is increasingly important in Earth science modeling Understand multi-component models with many options and parameters Distinguish simulations from one another Archive simulations with full descriptions for preservation and defensibility Enable run reproducibility ESMF Attributes store, aggregate, and output metadata Structured as name-value pairs Arranged in packages by purpose and convention: International Standards Organization (ISO), Climate and Forecast (CF), METAFOR Common Information Model (CIM), etc. Output in XML and other formats ESMF Attributes in CESM CESM components (land, atm, ice, ocean, sea ice) and driver are being instrumented with Attributes Information captured includes component level, field level, and system level metadata 05/11/2012 XSEDE Science Gateway

CESM Metadata in the ESG Gateway Screenshot of CESM metadata Implemented in the model by ESMF and Curator. Exported by CESM at run time. Uploaded to Purdue’s atom feed. Ingested and displayed by ESG. 05/11/2012 XSEDE Science Gateway

Data Discovery, Access and Analysis ESG: Science Gateway for Earth System Data Access Data discovery and access across federation of data nodes. Identity services and project specific access control. Data publishing services. Versioning and provenance support. Bulk data download, wget scripts. Data use tracking and metrics. Model documentation metadata browsing. ATOM CIM feed and ESG ingest. NCL Backend to Product Services Integrated 64-bit NCL visualization engine. Big data capable analysis functions. ParNCL as a drop-in replacement, for parallel computation. Plug-in component to NOAA/PMEL LAS Product Server. 05/11/2012 XSEDE Science Gateway

Demo 05/11/2012 XSEDE Science Gateway

Demo 05/11/2012 XSEDE Science Gateway

Demo 05/11/2012 XSEDE Science Gateway

Next Steps Transition into XD Release CESM portal Work with faculty and instructors to use in teaching Need help in connecting with users Federated authentication/identity between CESM and ESG gateways Making a rich set of tools for analysis and visualization available to CESM gateway users Add other resources to backend for better user experience 05/11/2012 XSEDE Science Gateway