1. www.esmf.ucar.edu Enhancements for Hydrological Modeling in ESMF Cecelia DeLuca/NCAR (ESMF) December 19, 2008 AGU Fall Meeting

2. www.esmf.ucar.edu Outline ESMF overview Results and projects relevant to the hydrological community ESMF + data services and semantic technologies A peek at the future

3. www.esmf.ucar.edu Earth System Modeling Framework (ESMF) Component-based framework that targets tightly coupled, computationally demanding weather and climate applications written in Fortran ESMF provides standard interfaces for model components ESMF provides common utilities and tools for routine modeling functions such as regridding between components Status Initiated by NASA in 2002 and developed and managed by a multi-agency consortium More than 76 ESMF components in the community, used by the NASA Modeling, Analysis and Prediction Program, the Department of Defense Battlespace Environments Institute, the National Weather Service, and many others

4. www.esmf.ucar.edu Each box is an ESMF gridded component or coupler component ESMF State objects carry data between components Every component (including couplers) has a standard interface to facilitate exchanges The ESMF architecture enables the assembly of many different systems ESMF Application Example Part of the GEOS-5 Atmospheric General Circulation Modeling system

5. www.esmf.ucar.edu Interoperability and standardization of interfaces There are only three ESMF component methods: Initialize, Run, and Finalize (I/R/F) Users associate their native code I/R/F methods with an ESMF Component through a registration process I/R/F methods cascade down the tree Small set of standard arguments: call ESMF_CompRun (myComp, importState, exportState, clock, phase, blockingFlag, rc)

6. www.esmf.ucar.edu Outline ESMF overview Results and projects relevant to the hydrological community ESMF + data services and semantic technologies A peek at the future

7. www.esmf.ucar.edu Data representation options 1. Representation in index space (Arrays) One or more logically rectangular tiles store indices and topology Sparse matrix multiply for regridding with user supplied interpolation weights Very general data representation, but limited interoperability since not much semantic info is encoded by the framework 2. Representation in physical space (Fields) Built on Arrays + some type of Grid Regridding using parallel on-line interpolation weight generation, bilinear or higher order Standard high-level data structures increase interoperability Supported Array distributions

8. www.esmf.ucar.edu Coupling options in ESMF Lots of flexibility in coupling approaches Single executable Multiple executable Coupling communications can be set up and called in a coupler Coupling communications can be setup in a coupler but called directly from within components – useful when it is inconvenient to return from a component in order to perform a coupling operation Recursive components for nesting higher resolution regions Ensemble management, including sequential execution of ensemble members Comp AComp B Comp A Comp B Contributed by U Maryland Coupler Array send/recv Multiple executable options

9. www.esmf.ucar.edu New Features and Applications The Flow-Following Finite Volume Icosahedral Model (FIM) from NOAA GSD is converting to ESMF to couple to National Weather Service models ESMF v3.1.1 beta release (September 2008) includes ◦ unstructured mesh class ◦observational data streams ◦on-line parallel regridding ◦standard metadata packets Consortium of U.S. operational and research centers (NOAA, NASA, DoD) has entered pilot phase of development of multi-model global ensemble using ESMF components ESMF parallel regrid: Grid-Grid, Grid-Mesh, Mesh-Mesh Bilinear and higher order National Unified Operational Prediction Capability

10. www.esmf.ucar.edu Targeting greater interoperability ESMF is a toolkit with many options for coupling, and ESMF compliance alone has not meant plug-and-play Several groups have been creating additional guidelines, conventions, and constraints for increased interoperability Example: a NUOPC operational consortium committee has reached detailed agreements on the following: ◦Componentization level - atm, ocean, land, ice, atm physics and dynamics ◦Clock - how time is passed between models and modules ◦Ownership – what models and modules are allowed to modify data held in common (e.g. aerosol species and concentrations) ◦Level of data representation – index space or physical space ◦Metadata – the description of the data, grid, clock, etc. ◦Coupling approach - NUOPC will provide a standard template. ◦Discussions are underway on IO, post-processing, and component invocation points

11. www.esmf.ucar.edu Parallel weight calculation and regrid: Results for ESMF in CCSM Interp. noise Interpolation noise in the derivative of the zonal wind stress grid index in latitudinal direction dTAUx/dy - closely related to the curl of the wind stress, which drives the upper ocean circulation and is therefore important to represent accurately in climate simulations Mapped from the T62 (2-degree) Community Atmospheric Model (CAM) grid to the gx1v5 (384x320 irregularly spaced) Noise is calculated as deviation of a point from the sum of itself plus four neighbors 33% reduction in noise globally compared to previous CCSM bilinear approach CCSM is now using ESMF for their standard remapping Black = bilinear Red = higher-order ESMF v3.1.1 Green = higher order ESMF v4.0.0

12. www.esmf.ucar.edu Performance Portablity 30+ platform/compiler combinations regression tested nightly, new ports include gfortran (4.3), Solaris, and (almost) Windows Performance at the petascale… The chart at right shows scaling of the ESMF sparse matrix multiply, used in regridding transformations, out to 16K processors. (ESMF v3.1.0rp2) Plot from Peggy Li, NASA/JPL Tested on ORNL XT4 in a variety of configurations. -N1 means 1 core per node. ESMF Sparse Matrix Multiply Run-Time

13. www.esmf.ucar.edu Outline ESMF overview Results and projects relevant to the hydrological community ESMF + data services and semantic technologies A peek at the future

14. www.esmf.ucar.edu Metadata export from ESMF (v4.0) ESMF holds information about grids, component inter-relationships Users can associate additional metadata with ESMF classes (Grids, Arrays, Fields, States, Components) as name/value pairs in Attribute packages Attribute packages can be standard (e.g CF conventions for Fields) or custom Packages can be nested – for example, metadata for an import State contains metadata for all the Fields in the State. Metadata can be exported in plain text or XML format … then ingested into portal, where users can search and browse components and datasets via the portal, “trackback” to the model configuration that created a dataset, create dynamic comparison tables for specific properties, create private or shared workspaces, etc. Excellent support for component and model inter-comparison projects. Image courtesy of Rocky Dunlap, Georgia Institute of technology The path from model metadata to web interface involves many format translations along the way

15. www.esmf.ucar.edu Numerical Techniques for Global Atmospheric Models Summer Colloquium Held at NCAR during summer 2008, about 80 participants 9 atmospheric dynamical core components archived Running 22 test cases at up to 5 resolutions Search, browse, trackback, comparison tables Automated upload of simulation data, publishing, and comparison Data search and browse via ESG Gateway Demonstrates how knowledge base can combines with ESMF http://dycore.ucar.edu Curator project sponsored by

16. www.esmf.ucar.edu Work in progress Translation of ESMF interfaces into web service interfaces to support loose coupling, multi-language coupling, and integration of components into service oriented architectures (Curator project) Integration of CCSM ESMF version with Kepler workflow tool (PhD student at Istanbul Technical University) Automated coupler generation (PhD student at Georgia Tech) New TeraGrid award: invoke CCSM ESMF version from an ESG-based science gateway on the TeraGrid - data and metadata from the run are automatically stored back to the portal to support intercomparison projects Working with Cray, Microsoft and NASA to finish port to Windows HPC 2008 Image courtesy of Rocky Dunlap, Georgia Institute of technology

17. www.esmf.ucar.edu A peek at the future … 2009 2010 2011 2012 2013 2014 2015 Interoperability Compliance definitions and verification tools Tutorial development and classroom deployment Integration with workflows Generation of service interfaces for loose coupling Metadata links to data and other services Automated model assembly and run environment Based on metadata files Numerical/computational maintenance and development