Components and Concurrency in ESMF Nancy Collins Community Meeting July 21, GMAO Seasonal Forecast NCAR/LANL CCSM NCEP Forecast GFDL FMS Suite MITgcm NASA GSFC GSI

Current Status Starting with ESMF release ESMF Components can be executed concurrently as well as serially. Child components can be created on a subset of the parent component’s PETs (new petList argument to the ESMF_ComponentCreate() call) The main loop looks very much the same, but a new Blocking flag can be set to return immediately on all PETs which do not run the component. They can then start another component. Ensemble runs are now supported as well.

Sequential vs Concurrent

ESMF_StateReconcile() New ESMF_State call ESMF Communications are all computed in parallel by each PET. Each PET computes independently what data needs to be sent and to who, and what data will be received and from whom. The reconcile process ensures all PETs have the global grid information for all data items, even those without local data. Then they can compute who they will be receiving data from, or to whom they need to send data.

Creating Ensembles Multiple Components running the same module code. Each can compute with a different set of input parameters.

Ensembles seem: But they are:

Get/SetInternalState Each component derived type maintains a private pointer to an allocated user data block. Think shared vs private variables in OpenMP, for example.

Futures: Threading Threads allow a single processor to have more than one stream of execution. Threading is enabled in the Child component at SetServices time by making a VM Set call.

Sequential Execution All PETs run the same component at the same time -- but each component is still computing in parallel.

Sequential Execution Details The Framework has a thin code layer between actual calls from one Component to the other. The Parent Component controls the execution sequencing.

Concurrent Execution Concurrency means different PETs are executing different Components.

Concurrent Details (1) The Parent Component decides how to allocate resources for the Child Components at Component Create time by giving each Child a list of PETs. It also controls the execution sequence of Child components.

Concurrent Details (2) The Framework tracks which PETs are supposed to run a Component. If the Component Run routine is called with the Blocking flag set to Non-Blocking, the Run call returns immediately on those PETS which should not be executing the Component.

Concurrent with Blocking If Component Run is called with the Blocking flag on, the red lines show where the Framework adds Barriers on all PETs to ensure the Component has finished before any Child returns.

Communications: Redist Basics Redistribution is used when the exact same grid is decomposed in different ways in different components. Running sequentially, the source, destination, and coupler all access the same data with no problems. Each PET computes the intersection between grids independently and stores the send and receive sizes and offsets.

Redist on same set of PETs The communication code in ESMF is a “me-centric” model. Each PET computes what data it needs to send and to who, and what data is expects to receive and from who. This scales well as number of processors grows. It does require each PET has the same information about the global grid and its decomposition.

Redist on Exclusive PETs What happens when the coupler tries to couple Components which do not run on exactly the same number of PETs, or exclusive PET sets.

After ESMF_StateReconcile() After calling ESMF_StateReconcile(), all PETs have the same global grid information, even if they contain no local data for that Field/Grid. Now the communication code can determine if it will be receiving data from a remote grid, or if it needs to send data to a remote grid.

Creating Ensembles Ensembles are running the same code with different parameter settings.

Ensemble Details ESMF ensembles running in the same executable look much like exclusive Components which do not communicate between themselves.

How Ensembles Seem What you might think you have when you create multiple components using the same module and entry points.

How Ensembles Are But in reality the two instances of the same component share both code and module global data. If one Component modifies the global data, all Components see the change.

SetInternalState at Init Time During Init, allocate a local derived type and store the pointer in the framework. There will be a separate pointer per instance of each component.

GetInternalState at Run Time During Run, retrieve the InternalState pointer from the framework. Using data inside that derived type, determine any per-instance settings, variables, local data, etc.

Futures: Transforms Transforms allow communication without returning to the calling Component. It weakens the isolation between Components by introducing communication sequencing dependencies.