NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH May 15, 2003 Nancy Collins, NCAR 2nd Community Meeting, Princeton, NJ Earth System Modeling Framework Superstructure NSIPP Seasonal Forecast NCAR/LANL CCSM NCEP Forecast GFDL FMS Suite MITgcm NASA GSFC PSAS

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH May 15, 2003 Superstructure Outline Superstructure Classes Design Principles Object Details Adoption Preparation

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH May 15, 2003 ESMF Class Structure MachineModel Layout DistGrid PhysGrid State Bundle Gridded Component Array Utilities: TimeMgr, Config, LogErr, I/O etc. F90 Superstructure Infrastructure Field Grid Regrid Comm Data Communications Route Coupler Component C++ Application Component

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH May 15, 2003 Superstructure Classes Gridded Component(s) Array Computation Data Application Component FieldBundleState Coupler Component(s)

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH May 15, 2003 Design Principles: Scalable Applications Since each ESMF application is also a component, entire ESMF applications may be treated as Gridded Components and nested within larger applications. climate_comp ocn_comp atm_comp ocn2atm_coupler atm_phys phys2dyn_coupler atm_dyn PE Example: atmospheric application containing multiple coupled components within a larger climate application

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH May 15, 2003 Design Principles: Local Communication All inter-component communication within ESMF is local. climate_comp ocn_comp atm_comp atm_phys phys2dyn_coupler atm_dyn PE atm2ocn _coupler This means: Coupler Components must be defined on the union of the PEs of all the components that they couple. In this example, in order to send data from the ocean component to the atmosphere, the Coupler mediates the send.

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH May 15, 2003 Design Principles: Import/Export States Gridded Components do not have access to the internals of other Gridded Components. They have 2 options for exchanging data with other Components; the first is to receive Import and Export States as arguments. coupler ocn_component subroutine ocn_run(comp, & ImportState,ExportState, Clock, rc) atm_component subroutine atm_run(comp, & ImportState,ExportState, Clock, rc) States contain flags for “is required”, “is valid”, “is ready”, etc.

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH May 15, 2003 Design Principles: Transforms and States Gridded Components using Transforms do not have to return control to a higher level component to send or receive State data from another component. They can receive function pointers which are methods that can be called on the states. coupler ocn_component call ESMF_StateXform(ex_state, & xform) atm_component call ESMF_StateXform(xform, & im_state) transform call ESMF_CompRun(atm, xform) call ESMF_CompRun(ocn, xform)

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH May 15, 2003 Design Principles: Registration Process Components must provide a single externally visible entry point which will register the other entry points with the Framework. Components can: - Register one or more Initialization, Run, Finalize, and Checkpoint entry points. - Register a private data block which can contain all data associated with this instantiation of the Component; particularly useful when running ensembles. Higher level Comp cmp_register() cmp_run() cmp_final() cmp_init() ESMF Framework Services Public subroutine Private subroutine

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH May 15, 2003 Gridded Component Computational model, data assimilation code Associated DELayout object Optional associated Grid Register entry point Import and Export States for: – Initialization phase – Run phase – Finalize phase

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH May 15, 2003 Coupler Component Data transformation code: – Regridding, Unit conversions, Conservation of quantities, Accumulation Register entry point Nested States for: – Initialization phase – Run phase – Finalize phase

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH May 15, 2003 Application Component Main program Container for single Component Creates Gridded Component Calls Registration routine Read configuration files, create Grid, create Main Clock, supply dummy Import/Export States for subcomponents. Single call to Init, Run, and Finalize routine

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH May 15, 2003 State Objects Annotated List of other ESMF objects Bundles, Fields, Arrays, and other States Flags for “needed”, “ready”, “valid” Named objects, CF conventions for Fields Import/Export States for Gridded Comps Single State for Coupler Comps

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH May 15, 2003 Transform Objects Concurrent Components can exchange States without returning to calling code Coupler’s role is to provide Transform object which contains subroutine pointer Subroutine transforms data from Export State of one component to Import State of another Needs higher level of control/sequencing

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH May 15, 2003 ESMF User Environments Languages: F90 (predominately), C++ Systems: Unix/Linux variants (inc IBM, SGI, Compaq, Linux clusters, Mac OS X) Memory: MPI, PThreads/SMP Execution: Sequential and Concurrent Runtime system: LoadLeveler, poe, bsub, mpirun

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH May 15, 2003 ESMF Component Characteristics Components can create other Components Nested parent/child relationship Annotated States contain all data passed between components Data objects use CF name conventions Can be created dynamically Initialization/Run/Finalize/Checkpoint

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH May 15, 2003 Adopting ESMF See ESMF User’s Guide, Section 12 Self-contained Components – Subroutines for major phases – States for Import/Export – Private data What results can be exported up to higher level applications which might be built

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH May 15, 2003 ESMF Adoption Curve: Incremental Path Minimum: Component interfaces/subroutines, Data in Fields, Time via Clocks, information exchange via States Plus any or all of: Time Mgr with Internal Clocks, Data in ESMF Fields/Grids, Regrid/Halo of Fields, Logging, Configuration files