NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH ESMF Infrastructure Layer 2 nd ESMF Community Meeting, GFDL, May 15 th

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH Talk Outline What is the infrastructure layer – What is it for – What it contains ESMF infrastructure in this release – ESMF version 1.0 function call † examples (code!!!) ( † aka API or methods) Next steps…. Infrastructure Utility Example

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH Infrastructure Layer 1.A standard software platform for enabling interoperability (developing couplers, ensuring performance portability). 2.Set of reusable software for Earth science applications. Streamlined development for researchers. NCAR Atmosphere GFDL Ocean NSIPP Land NCAR Atmosphere GFDL Ocean NSIPP Land My Sea Ice

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH Infrastructure Layer Scope Support Support for Physical Grids Regridding Decomposition/composition Communication Calendar and Time I/O Logging and Profiling ESMF Infrastructure User Code ESMF Superstructure

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH 2. ESMF provides a toolkit that components use to i.ensure interoperability ii.abstract common services Location Within ESMF Component: run(), checkpoint() Field: halo(), import(), export() + I/O Grid: regrid(), transpose() + Metrics Layout, PEList, Machine Model Application Component Gridded Components Coupler Components 1. ESMF provides an environment for assembling components. INFRASTRUCTURELAYER

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH Talk Outline What is the infrastructure layer – What is it for – What it contains ESMF infrastructure in this release – ESMF version 1.0 function call † examples (code!!!) ( † aka API or methods) Next steps…. Infrastructure utility example

Infrastructure Internal Organization MachineModel DELayout DistGrid PhysGrid Bundle Array Time, Alarm, Calendar, LogErr, I/O, Attributes F90 C++ Field Grid Regrid Comm Data Communications Route Two tiers Class hierarchy for data and communications

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH ESMF Infrastructure Fields and Grids Field Contains array, grid, array to grid mapping and metadata Array Holds actual data e.g. temperature, wind speed etc… Grid Contains physical and distributed grid Physical Grid Grid metrics, e.g. lat-lon coords, spacings, areas, volumes etc.. Distributed Grid Parallel decomposition information

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH Creating an ESMF Field 1.Fortran Array 2.Array Attach 3.Field Attach Array real, dimension(100,100) :: arr esArr = ESMF_ArrayCreate(arr,…) ESMF_Array info. ESMF_Array info. ESMF_FieldAttachArray(esFld,esArr,…) ESMF_Field metadata ESMF_Grid

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH Creating an ESMF Grid ESMF_Grid PhysGrid DistGrid call ESMF_PhysGridCreate…(…) call ESMF_DistGridCreate…(…)

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH Field and Grid Together : grid = ESMF_GridCreate(…,layout,…) : field_u = ESMF_FieldCreate(grid, array) : 1.Create field distributed over a set of decomposition elements (DE’s). 2.Domain decomposition determined by DELayout, layout. 3.Each object ( grid and field_u ) has internal representation. 4.Other parts of the infrastructure layer use the internal representation e.g. Regrid() – interpolation/extrapolation + redistribute over DE’s Redistribution() - general data rearrangement over DE’s Halo() – specialized redistribution ESMF_Array info. ESMF_Field metadata ESMF_Grid

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH Regrid Function mapping field’s array to a different physical and distributed grid. RegridCreate() – creates a Regrid structure to be used/re-used regrid = ESMF_RegridCreate(src_field, dst_field, method, [name], [rc]) Source, Dest field can be empty of field data (RegridCreate() uses grid metrics) – PhysGrid, DistGrid info used for setting up regrid – Resulting regrid can be used for other fields sharing same Grid Method specifies interpolation algorithm. For example, bilinear, b-spline, etc…

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH Regrid Interface (cont) RegridRun() – performs actual regridding call ESMF_RegridRun(src_field, dst_field, regrid,[rc]) – Communication and interpolation handled transparently. RegridDestroy() – frees up memory call ESMF_RegridDestroy(regrid,[rc]) src_field dst_field

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH Redistribution No interpolation or extrapolation Maps between distributed grids, field’s array and physical grid are the same i.e. Example: layout() created two distributed grids one decomposed in X and one decomposed in Y. – Redistribution() function maps array data between the distributions (a transpose/corner turn) – Communication handled transparently src_field dst_field

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH Halo Fields have distributed index space of – Exclusive {E}, compute {C} and local {L} region where Halo() fills points not in {E} or {C} from remote {E} e.g {E} {C} {L} {C} {L} {E} call ESMF_FieldHalo(field_foo, status) DE3 DE4

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH More functions in reference manual e.g

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH Bundle and Location Stream ESMF_Bundle – collection of fields on the same grid ESMF_LocationStream – Like a field but….. – unstructured index space with an associated physical grid space – useful for observations e.g. radiosonde, floats Functions for create(), regrid(), redistribute(), halo() etc…

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH ESMF Infrastructure Utilities Clock Alarm Calendar I/O Logging Profiling Attribute Machine model and comms Ensures consistent time between components Provides field level I/O in standard forms – netCDF, binary, HDF, GRIB, Bufr Consistent monitoring and messaging Consistent parameter handling Hardware and system software hiding. Platform customizable

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH Time Standard type for any component Calendar (support for range of calendars) ! initialize stop time to 13May2003, 2:00 pm call ESMF_TimeInit(inject_stop_time, & YR=int(2003,kind=ESMF_IKIND_I8), & MM=off_month, DD=off_day, H=off_hour, M=off_min, & S=int(0,kind=ESMF_IKIND_I8), & cal=gregorianCalendar, rc=rc) do while (currTime.le. inject_stop_time ) : call ESMF_ClockAdvance(localclock, rc=rc) call ESMF_ClockGetCurrTime(localclock, currtime, rc) end call ESMF_CalendarInit(gregorianCalendar, ESMF_CAL_GREGORIAN, rc)

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH I/O Field level binary, netCDF, HDF, GRIB, bufr – Currently I/O piped through 1 PE call ESMF_FieldAllGather(field_u, outarray, status) if (de_id.eq. 0) then write(filename, 20) "U_velocity", file_no call ESMF_ArrayWrite(outarray, filename=filename, rc=status) endif call ESMF_ArrayDestroy(outarray, status) – Current system fixed textual.

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH Attributes Flexible parameter specs, configuration e.g. file of parameters Code

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH Internal Classes Machine model – Captures system attributes, CPU, mem, connectivity graph – Useful for defining decomposition, load-balance, performance predictions. Comms – Communication driver, allows bindings to MPI, shared memory, vendor system libraries

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH Comms Performance Test Right mix (green) on Compaq gives x2 realized bandwidth (in large message limit)

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH Talk Outline What is the infrastructure layer – What is it for – What it contains ESMF infrastructure in this release – ESMF version 1.0 function call † examples (code!!!) ( † aka API or methods) Next steps…. Infrastructure utility example

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH Regrid Next Steps Support for all ESMF Grids Support for regridding methods: Bilinear Bicubic 1 st -order Conservative 2 nd -order Conservative Rasterized Conservative Nearest-neighbor distance-weighted average Spectral transforms 1-d interpolations (splines) Index-space (shifts, stencils) Adjoints of many above Halo Next Steps Support for periodicity More general haloing – in tandem with distributed grid evolution Adjoint forms

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH Distributed Grid Regular 2d already supported Next steps – Generalized 1d decomposition – Extend support for 2d and quasi regular decomposition – Spectral grid decompositions Larger set of metrics, grids High level routines for rapid definition of common grids. Physical Grid Next Steps

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH I/O Next Steps Broaden format set, binary, netCDF, HDF, GRIB, bufr Improve parallelization Full support for alarms Broader functionality Time/Logging/Profiling Next Steps

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH Summary ESMF Infrastructure Layer – Comprehensive class structure available in version 1.0 – Over the coming year significant extension of functionality will take place. – Feedback and comments on version 1.0 welcome

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH ESMF Infrastructure Utility detailed example Earl Schwab, ESMF Core Team, NCAR Time Manager

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH What is Time Manager? Clock for time simulation Time representation Time calculator Time comparisons Time queries F90 API, C++ implementation

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH Clock for time simulation type(ESMF_Clock) :: clock call ESMF_ClockInit(clock, timeStep, startTime, stopTime, rc=rc) do while (.not.ESMF_ClockIsStopTime(clock, rc)) ! Do application work. call ESMF_ClockAdvance(clock, rc=rc) end do

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH Clock (cont.) Clock queries/commands call ESMF_ClockGetCurrTime(clock, currTime, rc) call ESMF_ClockSetCurrTime(clock, currTime, rc) call ESMF_ClockGetTimeStep(clock, timeStep, rc) call ESMF_ClockSetTimeStep(clock, timeStep, rc) call ESMF_ClockGetAdvanceCount(clock, advanceCount, rc) call ESMF_ClockGetStartTime(clock, startTime, rc) call ESMF_ClockGetStopTime(clock, stopTime, rc) call ESMF_ClockGetPrevTime(clock, prevTime, rc) call ESMF_ClockSyncToWallClock(clock, rc)

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH Time Representation type(ESMF_Calendar) :: calendar1 call ESMF_CalendarInit(calendar1, ESMF_CAL_GREGORIAN, rc) - ESMF_CAL_GREGORIAN (3/1/-4800 to 10/29/292,277,019,914) - ESMF_CAL_JULIAN (+/- 106,751,991,167,300 days) - ESMF_CAL_NOLEAP - ESMF_CAL_360DAY

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH Time Representation (cont.) type(ESMF_Time) :: time1 call ESMF_TimeInit(time1, YR=int( 2003,kind=ESMF_IKIND_I8), & MM= 5, DD= 15, H= 15, cal=calendar1, rc=rc) - YR, MM, DD, H, M, S F90 optional - D, H, M, S arguments type(ESMF_TimeInterval) :: timeInterval1 call ESMF_TimeIntervalInit(timeInterval1, D=int( 90,kind=ESMF_IKIND_I8), rc=rc) - D, H, M, S F90 optional arguments

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH Time Calculator Time differencing Time increment/decrement by a time interval Time interval arithmetic (+, -, *, /)

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH Time Calculator (cont.) call ESMF_TimeInit(time1, YR=int( 2003,kind=ESMF_IKIND_I8), & MM= 5, DD= 15, cal=gregorianCalendar, rc=rc) call ESMF_TimeInit(time2, YR=int( 2003,kind=ESMF_IKIND_I8), & MM= 3, DD= 26, cal=gregorianCalendar, rc=rc) call ESMF_TimeIntervalInit(timeInterval1, D=int( 90,kind=ESMF_IKIND_I8, rc=rc) timeInterval2 = time2 - time1 time1 = time1 + timeInterval1 ! Uses F90 overloaded timeInterval3 = timeInterval1 * 2 ! operators double precision :: ratio ratio = timeInterval1 / timeInterval2

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH Time Comparisons >, =, F90 overloaded operators between any 2 times or time intervals if (time1 < time2) then … end if if (timeInterval1.ge. timeInterval2) then … end if

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH Time Queries call ESMF_TimeGet(time1, YR=yr, MM=mm, DD=dd, H=h, M=m, S=s, rc=rc) call ESMF_TimeIntervalGet(timeInterval1, D=d, H=h, M=m, S=s) call ESMF_TimeGetDayOfYear(time1, dayOfYear, rc)! double or integer call ESMF_TimeGetDayOfMonth(time1, dayOfMonth, rc) call ESMF_TimeGetDayOfWeek(time1, dayOfWeek, rc) call ESMF_TimeGetMidMonth(time1, midMonth, rc) call ESMF_TimeGetString(time1, string, rc) ! T12:20:19 (ISO 8601) call ESMF_TimeIntervalGetString(timeInterval1, string, rc) ! P1DT12H0M0S (ISO) call ESMF_TimeGetRealTime(time1, rc)

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH More information ESMF User’s Guide ESMF Reference Manual ESMF Requirements Document Time Manager F90 API & examples source code esmf_1_0_0_r/src/Infrastructure/TimeMgr/interface esmf_1_0_0_r/src/Infrastructure/TimeMgr/examples