1. NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 8 April 2003 V. Balaji / GFDL, Cecelia DeLuca / NCAR, Chris Hill / MIT, Max Suarez, Arlindo da Silva / NASA GSFC EGS - AGU Nice, France Architecture of the Earth System Modeling Framework NSIPP Seasonal Forecast NCAR/LANL CCSM NCEP Forecast GFDL FMS Suite MITgcm NASA GSFC PSAS

2. NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 8 April 2003 Outline ESMF Project Overview Related Projects and Interactions ESMF Design Overview ESMF Design Principles Timeline and Status

3. NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 8 April 2003 Technological Trends In climate research and NWP... increased emphasis on detailed representation of individual physical processes; requires many teams of specialists to contribute components to an overall modeling system. In computing technology... increase in hardware and software complexity in high- performance computing, as we shift toward the use of scalable computing architectures and open community software standards.

4. NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 8 April 2003 ESMF Project Overview GOAL: To increase software reuse, interoperability, ease of use and performance portability in climate, weather, and data assimilation applications PRODUCTS: Coupling superstructure and utility infrastructure software Synthetic code suite for validation and demonstration Set of 15 ESMF-compliant applications (including CCSM, WRF, GFDL models; MIT, NCEP and NASA data assimilation systems) Set of 8 interoperability experiments RESOURCES: $10.1M over 3 years from NASA Earth Science Technology Office

5. NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 8 April 2003 ESMF Collaborators NSF NCAR Tim Killeen, PI Byron Boville Cecelia DeLuca Roberta Johnson John Michalakes Al Kellie Jeff Anderson David Neckels Earl Schwab Nancy Collins Jon Wolfe Silverio Vasquez Dennis Flanigan Robbie Staufer DOE LANL Phil Jones NOAA NCEP Stephen Lord Mark Iredell Mike Young Weiyu Yang John Derber MIT John Marshall, PI Chris Hill NASA DAO Arlindo da Silva, PI Leonid Zaslavsky Will Sawyer Carlos Cruz University of Michigan Quentin Stout NOAA GFDL Ants Leetmaa V. Balaji Robert Hallberg Shep Smithline NASA NSIPP Max Suarez Michele Rienecker Christian Keppenne Atanas Trayanov DOE ANL Rob Jacob Jay Larson

6. NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 8 April 2003 ESMF Interoperability Demonstrations COUPLED CONFIGURATIONNEW SCIENCE ENABLED GFDL B-grid atm / MITgcm ocnGlobal biogeochemistry (CO2, O2), SI timescales. GFDL MOM4 / NCEP forecastNCEP seasonal forecasting system. NSIPP ocean / LANL CICESea ice model for extension of SI system to centennial time scales. NSIPP atm / DAO analysisAssimilated initial state for SI. DAO analysis / NCEP modelIntercomparison of systems for NASA/NOAA joint center for satellite data assimilation. DAO CAM-fv / NCEP analysisIntercomparison of systems for NASA/NOAA joint center for satellite data assimilation. NCAR CAM Eul / MITgcm ocnImproved climate predictive capability: climate sensitivity to large component interchange, optimized initial conditions. NCEP WRF / GFDL MOM4Development of hurricane prediction capability.

7. NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 8 April 2003 Outline ESMF Project Overview Related Projects and Interactions ESMF Design Overview ESMF Design Principles Timeline and Status

8. NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 8 April 2003 Larson/ANL DeLuca/NCAR-SCD Jones/LANL Stout/U Mich Killeen/NCAR Drake/ORNL Boville/NCAR-CGD Michalakes/NCAR-MMM Suarez/NASA Goddard Balaji/NOAA GFDL CCA WRF SciDAC CCSM GEMSFMS SWMF PRISM ESMF Related Projects ESMF Earth System Modeling Framework CCA DOE Common Component Architecture SciDAC DOE/NSF CCSM SciDAC Project GEMS Goddard Earth Modeling System FMS GFDL Flexible Modeling System SWMF Space Weather Modeling Framework WRF Weather Research and Forecast Model CCSM Community Climate System Model PRISM Program for Int. Earth System Modeling

9. NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 8 April 2003 ESMF and PRISM A European Earth system modeling infrastructure project, started December 2001 Funded by the European Commission ($4.8M) Involves current state-of-the-art atmosphere, ocean, sea-ice, atmospheric chemistry, land- surface and ocean-biogeochemistry models 22 partners: leading climate researchers and computer vendors, includes MPI, KNMI, UK Met Office, CERFACS, ECMWF, DMI.

10. NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 8 April 2003 ESMF and PRISM, cont. Working together to supplement CF convention for physical field names and quantities Component interface database developed by ESMF stores fields in model import and export states V. Balaji (GFDL) is ESMF liaison to PRISM

11. NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 8 April 2003 ESMF and CCA Common Component Architecture (CCA) is creating a minimal interface and sets of tools for linking high performance components. CCA can be used to implement frameworks and standards developed in specific domains (such as ESMF). DOE funded through SciDAC. Collaborators include LANL, ANL, LLNL, ORNL, Sandia, University of Tennessee, and many more. Ongoing ESMF collaboration with CCA/LANL on language interoperability. Shujia Zhou / Arlindo da Silva (NASA GSFC) currently prototyping an ESMF API using CCA tool CCAFFEINE; working towards CAM/DAO PSAS coupling using ESMF and CCA.

12. NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 8 April 2003 Outline ESMF Project Overview Related Projects and Interactions ESMF Design Overview ESMF Design Principles Timeline and Status

13. NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 8 April 2003 Architecture External Libraries Low Level Utilities Fields and Grids Layer Model Layer Coupling Layer ESMF Infrastructure User Code ESMF Superstructure BLAS, MPI, NetCDF, …

14. NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 8 April 2003 Field: Halo( ), Import( ), Export( ) + I/O ESMF Components 2.ESMF provides a toolkit that components use to i.increase interoperability ii.improve performance portability iii.abstract common services Component: Init( ), Run( ), Checkpoint( ) Grid: Regrid( ), Transpose( ) + Metrics Layout, PE List, Machine Model Application Component Gridded ComponentsCoupler Components 1.ESMF provides an environment for assembling geophysical components into an application.

15. NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 8 April 2003 General Features ESMF will be usable by models written in F90/C/C++ ESMF will be usable by models requiring adjoint capability ESMF will be usable by models requiring shared or distributed memory parallelism semantics ESMF will support SPMD and MPMD coupling ESMF will support several I/O formats, including GRIB/BUFR, netCDF, HDF ESMF will have uniform syntax across platforms ESMF will target a broad range of platforms, from major centers  commodity hardware

16. NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 8 April 2003 Outline ESMF Project Overview Related Projects and Interactions ESMF Design Overview ESMF Design Principles Timeline and Status

17. NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 8 April 2003 Design Principles: Scalable Applications Since each ESMF application is also a component, entire ESMF applications may be treated as Gridded Components and nested within larger applications. climate_comp ocn_comp atm_comp ocn2atm_coupler atm_phys phys2dyn_coupler atm_dyn PE Example: atmospheric application containing multiple coupled components within a larger climate application

18. NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 8 April 2003 Design Principles: Local Communication All inter-component communication within ESMF is local. climate_comp ocn_comp atm_comp atm_phys phys2dyn_coupler atm_dyn PE atm2ocn _coupler This means: Coupler Components must be defined on the union of the PEs of all the components that they couple. In this example, in order to send data from the ocean component to the atmosphere, the Coupler mediates the send.

19. NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 8 April 2003 Design Principles: Modularity Gridded Components don’t have access to the internals of other Gridded Components. Gridded Components may - pass their States through their argument list or - receive methods, through their argument list, for interacting with other Components (Transforms). coupler ocn_component call ESMF_StateXform(ex_state, & xform) atm_component call ESMF_StateXform(xform, & im_state) transform call ESMF_CompRun(atm, xform) call ESMF_CompRun(ocn, xform)

20. NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 8 April 2003 Design Principles: Uniform Communication API The same programming interface is used for shared memory, distributed memory, and combinations of both. Machine model = abstraction of machine architecture (num_nodes, num_pes_per_node) DE = a decomposition element - may be virtual, thread or MPI process Layout = an arrangement of DEs, in which dimensions requiring faster communication may be specified and resources arranged accordingly A 2 x 6 Layout of 4 3-processor nodes The data in a Grid is decomposed into the number of chunks specified in the Layout.

21. NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 8 April 2003 ESMF Class Structure MachineModel Layout DistGrid PhysGrid State Bundle Gridded Component Array Utilities: TimeMgr, LogErr, I/O etc. F90 Superstructure Infrastructure Field Grid Regrid Comm Data Communications More work Prototype exists Route Coupler Component C++

22. NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 8 April 2003 Outline ESMF Project Overview Related Projects and Interactions ESMF Design Overview ESMF Design Principles Timeline and Status

23. NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 8 April 2003 Timeline May 2002Draft Developer’s Guide and Requirements Document completed; Community Requirements Meeting and review held in D.C. July 2002ESMF VAlidation (EVA) suite assembled August 2002Architecture Document: major classes and their relationships; Implementation Report: language strategy and programming model; Software Build and Test Plan: sequencing and validation May 15 2003First API and software release, Community Meeting at GFDL July 2003First 3 interoperability experiments completed April 2004Second API and software release, Community Meeting July 2004All interoperability experiments completed Jan 2005Final release Currently planning for ongoing funding and extensions

24. NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 8 April 2003 May 2003 Release Focus for May 2003 ESMF release is on developing sufficient infrastructure and superstructure to achieve the initial set of interoperability experiments. These are: – FMS B-grid atmosphere coupled to MITgcm ocean – CAM atmosphere coupled to NCEP analysis – NSIPP atmosphere coupled to DAO analysis

25. NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 8 April 2003 More information 2nd ESMF Community Meeting at GFDL May 15, 2003 ESMF website: http://www.esmf.ucar.edu On the website: Information about the Community Meeting ESMF documents ESMF browsable source code repository On-line forms for community input and catalogued responses Milestone schedule, project archives and history, related projects Management plan, board and team contacts More …