NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 4 February 2003 Cecelia DeLuca, NCAR CCSM Software Engineering Working Group Mtg Boulder, CO Update on the Earth System Modeling Framework NSIPP Seasonal Forecast NCAR/LANL CCSM NCEP Forecast GFDL FMS Suite MITgcm

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 4 February 2003 Outline ESMF Project Overview Related Projects and Interactions ESMF Design Overview ESMF Design Principles Timeline and Status

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 4 February 2003 Technological Trends In climate research and NWP... increased emphasis on detailed representation of individual physical processes; requires many teams of specialists to contribute components to an overall modeling system. In computing technology... increase in hardware and software complexity in high- performance computing, as we shift toward the use of scalable computing architectures.

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 4 February 2003 ESMF Project Overview GOAL: To increase software reuse, interoperability, ease of use and performance portability in climate, weather, and data assimilation applications PRODUCTS: Coupling superstructure and utility infrastructure software Synthetic code suite for validation and demonstration Set of 15 ESMF-compliant applications (including CCSM, WRF, GFDL models; MIT, NCEP and NASA data assimilation systems) Set of 8 interoperability experiments RESOURCES: $10.1M over 3 years from NASA Earth Science Technology Office

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 4 February 2003 ESMF Collaborators NSF NCAR Tim Killeen, PI Byron Boville Cecelia DeLuca Roberta Johnson John Michalakes Al Kellie Jeff Anderson David Neckels Nancy Collins Jon Wolfe Earl Schwab NASA DAO Arlindo da Silva, PI Leonid Zaslavsky Will Sawyer Carlos Cruz NOAA NCEP Stephen Lord Mark Iredell Mike Young Wei Yu John Derber University of Michigan Quentin Stout MIT John Marshall, PI Chris Hill DOE Los Alamos National Lab Phil Jones NOAA GFDL Ants Leetmaa V. Balaji Robert Hallberg NASA NSIPP Max Suarez Michele Rienecker Christian Keppenne Atanas Trayanov DOE ANL Rob Jacob Jay Larson

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 4 February 2003 ESMF Interoperability Demonstrations COUPLED CONFIGURATIONNEW SCIENCE ENABLED GFDL B-grid atm / MITgcm ocnGlobal biogeochemistry (CO2, O2), SI timescales. GFDL MOM4 / NCEP forecastNCEP seasonal forecasting system. NSIPP ocean / LANL CICESea ice model for extension of SI system to centennial time scales. NSIPP atm / DAO analysisAssimilated initial state for SI. DAO analysis / NCEP modelIntercomparison of systems for NASA/NOAA joint center for satellite data assimilation. DAO CAM-fv / NCEP analysisIntercomparison of systems for NASA/NOAA joint center for satellite data assimilation. NCAR CAM Eul / MITgcm ocnImproved climate predictive capability: climate sensitivity to large component interchange, optimized initial conditions. NCEP WRF / GFDL MOM4Development of hurricane prediction capability.

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 4 February 2003 Outline ESMF Project Overview Related Projects and Interactions ESMF Design Overview ESMF Design Principles Timeline and Status

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 4 February 2003 Common Component Architecture Space Weather (SWMF), Geophysics, Ecosystem Dynamics, … CCSM Coupled System, NASA GEMS, GFDL FMS Single Earth system model or modeling system Earth System Modeling Framework Climate, Weather, Data Assimilation All Earth, Space, and Life Applications All HPC Applications Hierarchy of Modeling Infrastructure Projects

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 4 February 2003 Larson/ANL DeLuca/NCAR-SCD Jones/LANL Stout/U Mich Killeen/NCAR Drake/ORNL Boville/NCAR-CGD Michalakes/NCAR-MMM Suarez/NASA Goddard Balaji/NOAA GFDL CCA WRF SciDAC CCSM GEMSFMS SWMF PRISM ESMF Related Projects ESMF Earth System Modeling Framework CCA DOE Common Component Architecture SciDAC DOE/NSF CCSM SciDAC Project GEMS Goddard Earth Modeling System FMS GFDL Flexible Modeling System SWMF Space Weather Modeling Framework WRF Weather Research and Forecast Model CCSM Community Climate System Model PRISM Program for Int. Earth System Modeling

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 4 February 2003 ESMF and CCA Common Component Architecture (CCA) is creating a minimal interface and sets of tools for linking high performance components CCA can be used to implement frameworks and standards developed in specific domains (such as ESMF). DOE funded through SciDAC Collaborators include LANL, ANL, LLNL, ORNL, Sandia, University of Tennessee, and many more

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 4 February 2003 ESMF and CCA, cont. Shujia Zhou / Arlindo da Silva (NASA GSFC) currently prototyping an ESMF API using CCA tool CCAFFEINE; working towards CAM/DAO PSAS coupling using ESMF and CCA Ongoing ESMF collaboration with CCA/LANL on language interoperability Joint CCA / ESMF COMTESS meeting January 2003 Nancy Collins (NCAR SCD) is ESMF liaison to CCA Prototype ESMF / CCA interface in CCAFFEINE display

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 4 February 2003 ESMF and PRISM A European Earth system modeling infrastructure project, started December 2001 Funded by the European Commission ($4.8M) Involves current state-of-the-art atmosphere, ocean, sea-ice, atmospheric chemistry, land- surface and ocean-biogeochemistry models 22 partners: leading climate researchers and computer vendors, includes MPI, KNMI, UK Met Office, CERFACS, ECMWF, DMI.

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 4 February 2003 ESMF and PRISM, cont. Working together to supplement CF convention for physical field names and quantities Component interface database developed by ESMF stores fields in model import and export states V. Balaji (GFDL) is ESMF liaison to PRISM

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 4 February 2003 Outline ESMF Project Overview Related Projects and Interactions ESMF Design Overview ESMF Design Principles Timeline and Status

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 4 February 2003 Architecture External Libraries Low Level Utilities Fields and Grids Layer Model Layer Coupling Layer ESMF Infrastructure User Code ESMF Superstructure BLAS, MPI, NetCDF, …

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 4 February 2003 Field: Halo(), Import(), Export() + I/O ESMF Components 2.ESMF provides a toolkit that components use to i.increase interoperability ii.improve performance portability iii.abstract common services Component: Init(), Run(), Checkpoint() Grid: Regrid(), Transpose() + Metrics Layout, PE List, Machine Model Application Component Gridded ComponentsCoupler Components 1.ESMF provides an environment for assembling geophysical components into an application.

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 4 February 2003 General Features ESMF will be usable by models written in F90/C/C++ ESMF will be usable by models requiring adjoint capability ESMF will be usable by models requiring shared or distributed memory parallelism semantics ESMF will support SPMD and MPMD coupling ESMF will support several I/O formats, including GRIB/BUFR, netCDF, HDF ESMF will have uniform syntax across platforms ESMF will target a broad range of platforms, from major centers  commodity hardware

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 4 February 2003 Outline ESMF Project Overview Related Projects and Interactions ESMF Design Overview ESMF Design Principles Timeline and Status

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 4 February 2003 Design Principles: Scalable Applications Since each ESMF application is also a component, entire ESMF applications may be treated as gridded components and nested within larger applications. climate_comp ocn_comp atm_comp ocn2atm_coupler atm_phys phys2dyn_coupler atm_dyn PE Example: atmospheric application containing multiple coupled components within a larger climate application

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 4 February 2003 Design Principles: Local Communication All inter-component communication within ESMF is local. climate_comp ocn_comp atm_comp atm_phys phys2dyn_coupler atm_dyn PE atm2ocn _coupler This means: Coupler components must be defined on the union of the PEs of all the components that they couple. In this example, in order to send data from the ocean component to the atmosphere, the coupler mediates the send.

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 4 February 2003 Design Principles: Modularity Gridded Components don’t have access to the internals of other Gridded Components. Gridded Components may - pass their States through their argument list or - receive methods, through their argument list, for interacting with other Components (Transforms). coupler ocn_component call ESMF_StateXform(ex_state, & xform) atm_component call ESMF_StateXform(xform, & im_state) transform call ESMF_CompRun(atm, xform) call ESMF_CompRun(ocn, xform)

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 4 February 2003 Design Principles: Uniform Communication API The same programming interface is used for shared memory, distributed memory, and combinations of both. Machine model = abstraction of machine architecture (num_nodes, num_pes_per_node) DE = a decomposition element - may be virtual, thread or MPI process Layout = an arrangement of DEs, in which dimensions requiring faster communication may be specified and resources arranged accordingly. A 2 x 6 Layout of 4 3-processor nodes The data in a Grid is decomposed into the number of chunks specified in the Layout.

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 4 February 2003 ESMF Class Structure MachineModel Layout DistGrid PhysGrid State Bundle Component Array Utilities: TimeMgr, LogErr, I/O etc. F90 C++ Superstructure Infrastructure Field Grid Regrid Comm DataCommunications To do More work Prototype exists Route

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 4 February 2003 Outline ESMF Project Overview Related Projects and Interactions ESMF Design Overview ESMF Design Principles Timeline and Status

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 4 February 2003 Timeline May 2002Draft Developer’s Guide and Requirements Document completed Community Requirements Meeting and review held in D.C. July 2002ESMF VAlidation (EVA) suite assembled August 2002Architecture Document: major classes and their relationships Implementation Report: language strategy and programming model Software Build and Test Plan: sequencing and validation May First API and software release, Community Meeting at GFDL July 2003First 3 interoperability experiments completed April 2004Second API and software release, Community Meeting July 2004All interoperability experiments completed

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 4 February 2003 Development Status ~8 FTEs, half in NCAR SCD, actively developing code System tests started in early January 2003 Newly updated ESMF Architecture Document, ESMF Requirements Document Build and Test plans collected from all collaborators as well as core team Continually updated Developer’s Guide with coding conventions, configuration management, QA procedures, more Pseudocode application-level interfaces developed by CCSM, NSIPP, NCEP, NASA DAO Documents, system tests, pseudocodes on-line, on ESMF website

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 4 February 2003 May 2003 Release Focus for May 2003 ESMF release is on developing sufficient infrastructure and superstructure to achieve the initial set of interoperability experiments. These are: – FMS B-grid atmosphere coupled to MITgcm ocean – CAM atmosphere coupled to NCEP analysis – NSIPP atmosphere coupled to DAO analysis

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 4 February 2003 More information ESMF website: On the website: ESMF documents ESMF browsable source code repository On-line forms for community input and catalogued responses Milestone schedule, project archives and history Related projects Management plan, board and team contacts More …