Session 3: NUOPC Interoperability Layer Instructors: Rocky Dunlap and Fei Liu NOAA Cooperative Institute for Research in Environmental Sciences University of Colorado, Boulder Training at NRL Monterey August 5-6, 2015

Two Day Overview Day 1 morning – Overview of ESMF and component-based modeling – Coding exercise: Run a single-component ESMF application Day 1 afternoon – ESMF distributed data classes and wrapping your model data types with ESMF data types – Regridding, LocStream – Coding exercise: Grid/Mesh construction Day 2 morning – Overview of NUOPC interoperability layer – Coding exercise: Run a NUOPC prototype application Day 2 afternoon – Questions and time for discussion of ESMF/NUOPC in COAMPS and other NRL modeling applications 2

Reading an ESMF Component subroutine SetServices(comp, rc) type(ESMF_GridComp) :: comp ! must not be optional integer, intent(out) :: rc ! must not be optional ! Set the entry points for standard ESMF Component methods call ESMF_GridCompSetEntryPoint(comp, ESMF_METHOD_INITIALIZE, & userRoutine=AtmInit, rc=rc) call ESMF_GridCompSetEntryPoint(comp, ESMF_METHOD_RUN, & userRoutine=AtmDyn, phase=1, rc=rc) call ESMF_GridCompSetEntryPoint(comp, ESMF_METHOD_RUN, & userRoutine=AtmPhys, phase=2, rc=rc) call ESMF_GridCompSetEntryPoint(comp, ESMF_METHOD_FINALIZE, & userRoutine=AtmFinal, rc=rc) rc = ESMF_SUCCESS end subroutine subroutine AtmDyn(comp, importState, exportState, clock, rc) type(ESMF_GridComp) :: comp type(ESMF_State) :: importState, exportState type(ESMF_Clock) :: clock integer :: rc call model_dyn() ! call the dynamics end subroutine 3

Reading an ESMF Driver program AtmDriver use ATM, only: ATM_SetServices => SetServices … call ESMF_GridCompSetServices(comp, userRoutine=ATM_SetServices, rc=rc) … call ESMF_GridCompInitialize(comp, importState=importState, & exportState=exportState, clock=clock, userRc=userrc, rc=rc) do while (.not. ESMF_ClockIsStopTime(clock)) call ESMF_GridCompRun(comp, phase=1, importState=importState, & exportState=exportState, clock=clock, userRc=userrc, rc=rc) call ESMF_GridCompRun(comp, phase=2, importState=importState, & exportState=exportState, clock=clock, userRc=userrc, rc=rc) … enddo call ESMF_GridCompFinalize(comp, importState=importState, & exportState=exportState, clock=clock, userRc=userrc, rc=rc) … end program 4

This Session… Understand the goals of the NUOPC Layer. Understand the purpose of each of the four generic components: Driver, Model, Connector, and Mediator and why they need to be “specialized” to provide application-specific code. Learn how NUOPC Components can be assembled into different configurations with customized run sequences. Learn about the “handshaking” steps in the initialization sequence and which ones require or allow user specialization. Review a set of recommended steps for converting an existing model component into a NUOPC application. 5

National Unified Operational Prediction Capability ESMF – provides common data types (field, grid, mesh, clock, etc.) and component interfaces (init, run, finalize) – BUT: there are few constraints on how to organize Components into an application and what happens at each model phase – RESULT: even if two independent models are built with ESMF types, interoperability is not guaranteed National Unified Operational Prediction Capability (NUOPC) Consortium of U.S. operational weather and water prediction centers – Both an agreement among agencies and a software layer implementation – Accelerate research to operations and model reuse – Technical goal: Increase interoperability of ESMF-based applications 6

NUOPC Interoperability Layer 7 NUOPC guarantees that Components provide: As in ESMF, a single, public entry point called SetServices A set of standard initialize phases that can be used to link fields across models and ensure model clock synchronization A standard run phase (or phases) that ensures that incoming and outgoing fields are consistent with the Component’s clock. A finalize method that deallocates memory and releases file handles. A makefile fragment with a small number of variables used for compiling and linking against the Component. ESMF API NUOPC API NUOPC is a layer on top of ESMF and is included with the ESMF distribution.

NUOPC Generic Components 8 Model : Implements a specific physical domain, e.g. atmosphere, ocean, wave, ice Mediator : Scientific coupling code (flux calculations, accumulation, averaging, etc.) between (potentially multiple) Models Connector : Connects pairs of components in one direction, e.g. Model to/from Model, or Model to/from Mediator Executes simple transforms (Regrid/Redist) Driver : Provides a harness for Models, Mediators, and Connectors (supporting hierarchies) Coordinates initialize and run sequences

Example Configurations of NUOPC Components 9 ab c a)A NUOPC Driver for two Models that share data via two Connectors b)A four-Model system with centralized Mediator. The OCN and WAVE components share data directly without going through the Mediator c)A complex system showing multiple Atmosphere components running as an ensemble. The highest level Mediator sees the ensemble as a single Model component.

Initialization Sequence The Initialize Phase Definition (IPD) regulates the interaction of NUOPC Components during their initialization process. The IPD ensures that: – Each Model advertises a set of required import and available export Fields, and that those Fields have standard names and associated units in the NUOPC Field Dictionary. – Each Connector identifies matching Fields between Models and establishes a connection and communication RouteHandle. – All Models have their import dependencies satisfied. – All Model Clock’s agree on the start time. – Two Models with connected Fields agree on who will provide the grid structure (can be both). – Model Fields are initialized at the right time and a timestamped according to the Model clock. – The Driver has established a run sequence. 10

Run Sequence The Run Sequence is set dynamically (during the initialize sequence) and controls the master time loop. Each Driver maintains a Run Sequence over its child Components. Run Sequences supported: – Simple loops in which all Components exchange data at the same coupling interval – Multiple timescales in which some Components communicate less frequently than others – Explicit, semi-implicit, and implicit modes in which some Components may run ahead or lag behind to satisfy numerical constraints 11

Sequential and Concurrent Components Drivers support both sequential and concurrent execution of child Components. The amount of concurrency is dependent on: – whether Components are assigned disjoint or overlapping PETs – data dependencies introduced by scientific constraints, e.g., the frequency of coupling required Drivers have the option of assigning a PET List when a Component is added 12

Specialization Specialization is a way of customizing the generic NUOPC Components to meet your application’s requirements. A specialization either: – provides an implementation not provided by NUOPC, or – overrides (replaces) a default behavior There are three ways to specialize a NUOPC Component: – by providing an entire initialize or run phase method – by registering a method at predefined specialization points – by setting or modifying metadata attributes 13

Some Specialization Points NUOPC_Driver – label_SetModelServices – Add Components to a Driver – label_SetRunSequence – Change default run sequence NUOPC_Model – label_SetClock – Modify incoming clock, e.g., to change timestep length – label_DataInitialize – Initialize Fields in Export State – label_Advance – Take a timestep NUOPC_Connector – label_ComputeRouteHandle – Store custom redist, regrid, etc. – label_ExecuteRouteHandle – Execute redist, regrid, etc. NUOPC_Mediator – label_CheckImport – Verify import fields – label_Advance – Execute mediation code 14

Knowing When to Specialize A key part of making a component NUOPC compliant is understanding the default behavior of the generic NUOPC components when it is necessary to specialize. In this next part, we’ll walk through the initialize sequence of a NUOPC application with two Models and two Connectors. This will give an idea of the basic control flow in a NUOPC application. Along the way, we’ll see important specialization points. 15

Walk through of 2-Model, Explicit Coupling Example 16 Architecture: One driver, two models, two connectors Coupling: Explicit time dependency requires coupling exchange at beginning of each coupling interval.

Driver 2-Model, Explicit Coupling 17 Driver ATM OCN Atm2Ocn Ocn2Atm start: 0 stop: 10 dt: 2 Init Phase 1* (Driver) (NUOPC provided) MainApp 1.Set internal clock to copy of incoming clock, if present. 2.Specialization: SetModelServices 3.Optional specialization: SetRunSequence 4.Drive initialization sequence for child Components (Models, Mediators, Drivers, Connectors). Run sequence: 1.Atm2Ocn 2.Ocn2Atm 3.ATM 4.OCN * Some details left out to simplify. Import:Export: Import:

2-Model, Explicit Coupling 18 Driver ATM OCN start: 0 stop: 10 dt: 2 Init Phase 2* (Model) (User provided) MainApp * Some details left out to simplify. 1.Advertise Fields in Import and Export States. “Advertising” a Field does not allocate memory, but allows each Model to announce the Standard Names of required and provided Fields. The NUOPC Field Dictionary is checked to verify that the advertised field names exist and have the correct units. Atm2Ocn Ocn2Atm Import: SST Export: DNRAD Export: SST Import: DNRAD Run sequence: 1.Atm2Ocn 2.Ocn2Atm 3.ATM 4.OCN

2-Model, Explicit Coupling 19 Driver ATM OCN start: 0 stop: 10 dt: 2 Init Phase 1* (Connector) (NUOPC provided) MainApp * Some details left out to simplify. 1.Construct the CplList attribute, which contains a list of Fields with matching Standard Names. These Fields are connected. Atm2Ocn Ocn2Atm Import: SST Export: DNRAD Export: SST Import: DNRAD CplList: SST CplList: DNRAD Run sequence: 1.Atm2Ocn 2.Ocn2Atm 3.ATM 4.OCN

2-Model, Explicit Coupling 20 Driver ATM OCN start: 0 stop: 10 dt: 2 Init Phase 2* (Model) (User provided) MainApp * Some details left out to simplify. 1.Realize connected Import and Export Fields. A connected field has a match in another Component. “Realizing” a Field means that a Grid is defined and memory is allocated. Fields that are not connected do not have to be realized. Atm2Ocn Ocn2Atm Import: SST Export: DNRAD Export: SST Import: DNRAD CplList: SST CplList: DNRAD Import: SST Export: DNRAD Export: SST Import: DNRAD Run sequence: 1.Atm2Ocn 2.Ocn2Atm 3.ATM 4.OCN

2-Model, Explicit Coupling 21 Driver ATM OCN start: 0 stop: 10 dt: 2 Init Phase 2* (Connector) (NUOPC provided) MainApp * Some details left out to simplify. 1.Construct srcFields and dstFields FieldBundles. Atm2Ocn Ocn2Atm Import: SST Export: DNRAD Export: SST Import: DNRAD RH: Bilinear Regrid Import: SST Export: DNRAD Export: SST Import: DNRAD 2.Optional specialization: ComputeRouteHandle 3.If not specialized, compute default RouteHandle as bilinear Regrid between srcFields and dstFields. Drivers also have the ability to modify CplList attributes to change connection options. RH: Bilinear Regrid Run sequence: 1.Atm2Ocn 2.Ocn2Atm 3.ATM 4.OCN

2-Model, Explicit Coupling 22 Driver ATM OCN start: 0 stop: 10 dt: 2 Init Phase 3* (Model) (NUOPC provided) MainApp * Some details left out to simplify. 1.Set internal clock as copy of incoming clock. Atm2Ocn Ocn2Atm Import: SST Export: DNRAD Export: SST Import: DNRAD RH: Bilinear Regrid Import: SST Export: DNRAD Export: SST Import: DNRAD 2.Optional specialization: SetClock (adjust timestep, set Alarms) RH: Bilinear Regrid start: 0 stop: 10 dt: 2 start: 0 stop: 10 dt: 2 3.Ensure all Import Fields are connected, raising an error if not. start: 0 stop: 10 dt: 1 Run sequence: 1.Atm2Ocn 2.Ocn2Atm 3.ATM 4.OCN

2-Model, Explicit Coupling 23 Driver ATM OCN start: 0 stop: 10 dt: 2 Init Phase 4* (Model) (NUOPC provided) MainApp * Some details left out to simplify. 1.Optional specialization: Data Initialize (Fields in Export State). Atm2Ocn Ocn2Atm Import: SST Export: DNRAD Export: SST Import: DNRAD RH: Bilinear Regrid Import: SST Export: DNRAD Export: SST Import: DNRAD 2.Timestamp Fields in export state to current time on internal clock. RH: Bilinear Regrid start: 0 stop: 10 dt: 2 start: 0 stop: 10 dt: 2 start: 0 stop: 10 dt: 1 Export: DNRAD[0] Export: SST[0] Run sequence: 1.Atm2Ocn 2.Ocn2Atm 3.ATM 4.OCN

2-Model, Explicit Coupling 24 Driver ATM OCN start: 0 stop: 10 dt: 2 Run* (Driver) (NUOPC provided) MainApp * Some details left out to simplify. 1.Override specialization: SetRunClock. Default is to set internal clock stop time to one timestep on incoming clock, if present. Atm2Ocn Ocn2Atm Import: SST Export: DNRAD Export: SST Import: DNRAD RH: Bilinear Regrid Import: SST Export: DNRAD Export: SST Import: DNRAD 2.Drive timestepping loop from current time to stop time (internal clock), calling child Components according to Run Sequence. RH: Bilinear Regrid start: 0 stop: 10 dt: 2 start: 0 stop: 10 dt: 2 start: 0 stop: 10 dt: 1 Export: DNRAD[0] Export: SST[0] Run sequence: 1.Atm2Ocn 2.Ocn2Atm 3.ATM 4.OCN

2-Model, Explicit Coupling 25 Driver ATM OCN start: 0 stop: 10 dt: 2 Run* (Connector) (NUOPC provided) MainApp * Some details left out to simplify. Atm2Ocn Ocn2Atm Import: SST Export: DNRAD Export: SST Import: DNRAD RH: Bilinear Regrid Import: SST Export: DNRAD Export: SST Import: DNRAD 2.Default is to execute pre-computed RouteHandle between srcFields and dstFields RH: Bilinear Regrid start: 0 stop: 10 dt: 2 start: 0 stop: 10 dt: 1 Export: DNRAD[0] Export: SST[0] 1.Optional specialization: ExecuteRouteHandle. Run sequence: 1.Atm2Ocn 2.Ocn2Atm 3.ATM 4.OCN 3.Update timestamp on dstFields to match srcFields. Import: SST[0] Import: DNRAD[0]

2-Model, Explicit Coupling 26 Driver ATM OCN curr: 0 stop: 10 dt: 2 Run* (Model) (NUOPC provided) MainApp * Some details left out to simplify. Atm2Ocn Ocn2Atm Import: SST Export: DNRAD Export: SST Import: DNRAD RH: Bilinear Regrid Import: SST[0] Export: DNRAD Export: SST Import: DNRAD[0] 2.Override specialization: CheckImport. Default is to verify all Import Fields are at current time. RH: Bilinear Regrid curr: 0 stop: 10 dt: 2 curr: 0 stop: 10 dt: 1 Export: DNRAD[0] Export: SST[0] 1.Override specialization: SetRunClock. Default is to set internal clock stop time to one timestep on incoming clock, if present. curr: 0 stop: 2 dt: 1 curr: 0 stop: 2 dt: 2 Run sequence: 1.Atm2Ocn 2.Ocn2Atm 3.ATM 4.OCN 3.Timestepping loop calls Advance specialization until stop time. Export Fields timestamped. curr: 1 stop: 2 dt: 1 Export: DNRAD[1] curr: 2 stop: 2 dt: 1 Export: DNRAD[2] curr: 2 stop: 2 dt: 2 Export: SST[2]

2-Model, Explicit Coupling 27 Driver ATM OCN curr: 0 stop: 10 dt: 2 Run* (Driver) (NUOPC provided) MainApp * Some details left out to simplify. Atm2Ocn Ocn2Atm Import: SST Export: DNRAD Export: SST Import: DNRAD RH: Bilinear Regrid Import: SST[0] Export: DNRAD Export: SST Import: DNRAD[0] RH: Bilinear Regrid curr: 0 stop: 10 dt: 2 curr: 0 stop: 10 dt: 1 Export: DNRAD[0] Export: SST[0] 1.Driver advances its internal clock at end of run sequence. The run sequence is repeated by the Driver until its internal clock reaches the stop time. curr: 0 stop: 2 dt: 1 curr: 0 stop: 2 dt: 2 Run sequence: 1.Atm2Ocn 2.Ocn2Atm 3.ATM 4.OCN curr: 1 stop: 2 dt: 1 Export: DNRAD[1] curr: 2 stop: 2 dt: 1 Export: DNRAD[2] curr: 2 stop: 2 dt: 2 Export: SST[2] curr: 2 stop: 10 dt: 2

2-Model, Explicit Coupling 28 Driver ATM OCN curr: 2 stop: 10 dt: 2 Run* (Connector) (NUOPC provided) MainApp * Some details left out to simplify. Atm2Ocn Ocn2Atm Import: SST Export: DNRAD Export: SST Import: DNRAD RH: Bilinear Regrid Import: SST[0] Export: DNRAD Export: SST Import: DNRAD[0] RH: Bilinear Regrid Export: DNRAD[0] Export: SST[0] Run sequence: 1.Atm2Ocn 2.Ocn2Atm 3.ATM 4.OCN Export: DNRAD[1] curr: 2 stop: 2 dt: 1 curr: 2 stop: 2 dt: 2 Export: DNRAD[2] Export: SST[2] 2.Default is to execute pre-computed RouteHandle between srcFields and dstFields 1.Optional specialization: ExecuteRouteHandle. 3.Update timestamp on dstFields to match srcFields. Import: SST[2] Import: DNRAD[2]

29 Some time later…

2-Model, Explicit Coupling 30 Driver ATM OCN curr: 10 stop: 10 dt: 2 Finalize* (Driver) (NUOPC provided) MainApp * Some details left out to simplify. Atm2Ocn Ocn2Atm RH: Bilinear Regrid RH: Bilinear Regrid Run sequence: 1.Atm2Ocn 2.Ocn2Atm 3.ATM 4.OCN Export: DNRAD[10] curr: 10 stop: 10 dt: 1 curr: 10 stop: 10 dt: 2 Export: SST[10] 2.Optional specialization: Finalize (Driver itself) 1.Execute Finalize() on all child Components. (This can be specialized.) 3.Destroy Models and their States followed by Connectors. Internal cleanup. Import: SST[8] Import: DNRAD[8] Import: Export: Import:

Making an Existing Model Component NUOPC Compliant 1.Create a NUOPC Model “cap” starting from a basic template. 2.Do not try to implement the entire initialize sequencing up front – instead, call your model’s existing initialize subroutine and provide stubs for any required NUOPC initialize phases. 3.Call your model’s update routine from the Advance specialization. If possible, use an offline mode (read from file) since the model will run in uncoupled mode. 4.Plug your Model “cap” into a generic NUOPC Driver and try to get through the run sequence. 5.Then, go back and split up the initialize phases as required by NUOPC. During the realize phase, you will need to create an ESMF_Grid and ESMF_Field s and hook them into your model variables. 6.Test the “cap” in coupled mode by adding another Model and Connectors to the generic Driver. 31

NUOPC Tools: Compliance Checker The Compliance Checker is a run-time analysis tool provides detailed log output about the interaction of NUOPC components. It is turned on a run time by setting the environment variable: $ export ESMF_RUNTIME_COMPLIANCECHECK=ON It reports on: which init, run, finalize phases are registered timekeeping fields passed between components via import/export states component-, state-, and field-level metadata 32

Sample Compliance Checker Output INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM:>START register compliance check INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM: phase Zero for Initialize registered INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM: 5 phase(s) of Initialize registered INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM: 1 phase(s) of Run registered INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM: 1 phase(s) of Finalize registered INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM:>STOP register compliance check INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM2MED:>START register compliance check INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM2MED: phase Zero for Initialize registered INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM2MED: 3 phase(s) of Initialize registered INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM2MED: 1 phase(s) of Run registered INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM2MED: 1 phase(s) of Finalize registered INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM2MED:>STOP register compliance check INFO PET0 COMPLIANCECHECKER:|->|->|->:MED2ATM:>START register compliance check INFO PET0 COMPLIANCECHECKER:|->|->|->:MED2ATM: phase Zero for Initialize registered INFO PET0 COMPLIANCECHECKER:|->|->|->:MED2ATM: 3 phase(s) of Initialize registered INFO PET0 COMPLIANCECHECKER:|->|->|->:MED2ATM: 1 phase(s) of Run registered INFO PET0 COMPLIANCECHECKER:|->|->|->:MED2ATM: 1 phase(s) of Finalize registered INFO PET0 COMPLIANCECHECKER:|->|->|->:MED2ATM:>STOP register compliance check INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM: >START InitializePrologue for phase= INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM: importState name: modelComp 1 Import State INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM: importState stateintent: ESMF_STATEINTENT_IMPORT INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM: importState itemCount: INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM: exportState name: modelComp 1 Export State INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM: exportState stateintent: ESMF_STATEINTENT_EXPORT INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM: exportState itemCount: 0

NUOPC Tools: Component Explorer 34 The Component Explorer is a run-time tool that can be used to gain insight into a NUOPC Component. It acts like a NUOPC Driver, but reports component information to standard out. A configuration file ( explorer.config ) is used to set up the clock and has options for whether or not to enter the run phase. (Most relevant information is available during the initialize sequence.) $./nuopcExplorerScript $ mpirun -np X./nuopcExplorerApp

Component Explorer Output 35 NUOPC Component Explorer App Exploring a component with a Fortran module front... Model component # 1 InitializePhaseMap: IPDv00p1=1 IPDv00p2=2 IPDv00p3=3 IPDv00p4=4 Model component # 1 // name = ocnA ocnA: : Attribute is present but NOT set! ocnA: importState // itemCount = 2 ocnA: importState // item # 001 // [FIELD] name = pmsl = air_pressure_at_sea_level = Pa = Air Pressure at Sea Level = pmsl ocnA: importState // item # 002 // [FIELD] name = rsns = surface_net_downward_shortwave_flux = W m-2 = Surface Net Downward Shortwave Flux = rsns ocnA: exportState // itemCount = 1 ocnA: exportState // item # 001 // [FIELD] name = sst = sea_surface_temperature = K = Sea Surface Temperature = sst

NUOPC Tools: Cupid Cupid is a static analysis and code generation tool the provides guidance to developers working with NUOPC application code. 36

Understand what’s there: Reverse Engineer an ESMF/NUOPC Application

Check for issues: Static Compliance Checking NUOPC View compact outline of NUOPC component source code linked with source code editor contextual reference documentation compliance issues shown in red

Write compliant code: In-place Code Generation

NUOPC Resources Website: Reference Manual: Prototype Codes description: Prototype Codes repository: MF_7_0_0_beta_snapshot_50/ MF_7_0_0_beta_snapshot_50/ Technical support 40

41

Extra Slides 42

NUOPC Generic Components 43 ESMF_GridCompESMF_CplComp NUOPC_Driver NUOPC_ModelNUOPC_MediatorNUOPC_Connector NUOPC Generic Components are specialized versions of GridComp and CplComp.