The Earth System Modeling Framework Robert Oehmke, Gerhard Theurich, Cecelia DeLuca NOAA Cooperative Institute for Research in Environmental Sciences University of Colorado, Boulder Gung Ho Meeting Bath, England November 2, 2014

Outline ESMF Overview ESMF Infrastructure ESMF Superstructure New Directions: NUOPC/ESPS and Cupid Current Status and Future Work

Motivation In climate research and numerical weather prediction… increased emphasis on detailed representation of individual physical processes; requires many teams of specialists to contribute components to an overall “coupled” modeling system (e.g. atmosphere, ocean, sea ice) In computing technology... increase in hardware and software complexity in high-performance computing, as we shift toward the use of scalable and now accelerator-based computing architectures In software … emergence of frameworks to promote code reuse and interoperability The ESMF is a focused community effort to tame the complexity of models and the computing environment. It leverages, unifies and extends existing software frameworks, creating new opportunities for scientific contribution and collaboration.

What is ESMF? ESMF provides: – Tools for building models – Tools for coupling models ESMF is: – Portable: Unix/Linux and Windows (Cygwin/MinGW) systems Tested on >40 different OS/Compiler/MPI combinations every night – Well tested: More than 6000 unit/system tests – Parallel: Based on MPI (with “mpiuni” bypass mode available) OpenMP and Pthreads support – Flexible: Interfaces to multiple languages A wide range of functionality and supported options

Interfaces Complete F95 API: – use ESMF – Derived types and methods – Investigating moving to Fortran 2003 Limited C API: – #include “ESMC.h” – Structs and methods Limited Python API: – Import ESMPy – Classes with methods Applications: – File-based interpolation weight generation: mpirun –np ESMF_RegridWeightGen –s …. – File-based weight generation AND application of weights: mpirun –np ESMF_Regrid (coming next release)

The ESMF Sandwich Superstructure: Component data structures and methods for coupling model components Infrastructure: Field data structures and methods for building model components, and utilities for coupling YOU DON’T NEED TO USE THEM BOTH!

Outline ESMF Overview ESMF Infrastructure ESMF Superstructure New Directions: NUOPC/ESPS and Cupid Current Status and Future Work

ESMF Infrastructure Distributed data classes: Used to hold data spread across a set of processors – Represents data so ESMF can perform operations on it – Provides a standard representation to be passed between components – Can reference user memory (usually) – Consists of two kinds of structures: Index Space classes (Arrays) Physical Space classes (Fields) Utilities: – Time Manager: Classes to represent time, time intervals, alarms… Used in ESMF for passing time info between models, time loops, etc. Also useful for doing calculations with time, conversions, etc. – Attributes: Allow metadata to be attached to ESMF classes Instrument models to be more self describing Can be written to various file formats, e.g. CIM compliant XML – Others: Logging (LogError), Virtual Machine (VM), Config

Index Space Distributed Data Distgrid: Represents index space and distribution across processors – Supports multiple index space representations From arbitrary 1-D sequences To N-D Tiles To N-D Tiles connected together along their edges – Supports multiple distribution options Array: A distributed index space data container – Array = data + Distgrid – Supports different data types: integer 4, real 4, real 8, … – Other options: halos, undistributed dimensions, … ArrayBundle: A set of Arrays which can be operated on at one time.

Physical Space Grid Representation Classes Grid: – Structured representation of a region – Consists of logically rectangular tiles Mesh: – Unstructured representation of a region – In 2D: polygons with any number of sides N-gon support added for MetOffice users – In 3D: tetrahedrons & hexahedrons Xgrid (Exchange Grid): – Represents boundary layer between two regions – Represented by custom constructed Mesh LocStream (Location Stream): – Set of disconnected points – E.g. locations of observations

Physical Space Distributed Data Field: A distributed physical space data container – Field = data + grid representation class (e.g. Grid, Mesh, …) – Based on Array/Distgrid, so supports those index/distribution options – Can get corresponding coordinates from grid representation class – Supports different data types: integer 4, real 4, real 8, … – Other options: halos, undistributed dimensions, … FieldBundle: A set of Fields which can be operated on at one time.

Distributed Data Class Operations Sparse Matrix Multiply: – Apply coefficients (weights) in parallel to distributed data – Highly tuned for efficiency/auto-tunes for optimal execution – Underlies most of ESMF distributed data operations Redistribution: – Move data between distributions without changing values – Useful in cases where grid doesn’t change, but distribution does Halo: – Fill “Halo” cells which hold data from another processor – Useful during computations on distributed data Regridding: – Move data from one grid to a different one – Useful when moving data between models with different grids – Only available on physical space data classes

Basic Computation ! Create grid representation class grid=ESMF_GridCreate(….) ! Create Field on grid representation class Field=ESMF_FieldCreate(grid,….) ! Create Halo Communication Structure Call ESMF_FieldHaloStore(Field,…,routehandle) ! Loop over time do t=1,…. ! Get Field data Call ESMF_FieldGet(Field, farrayPtr=ptr_to_data,…) ! Loop over memory doing computations on data do i=1,… do j=1… ptr_to_data(i,j)=…. enddo ! Update halo Call ESMF_FieldHalo(Field,…) enddo

Sparse Matrix Performance Example Performance of sparse matrix multiply in CESM (1_1_beta09) CPL7 Run on Cray XK6 (jaguarpf)

Regrid: Features Interfaces: F90, C, Python, ESMF_RegridWeightGen (file-based separate application) Multiple interpolation types: – Bilinear – Higher order patch recovery Yields better derivatives/smoother results than bilinear Based on “patch recovery” used in finite element modeling [1][2] – Nearest neighbor – First order conservative Path between points in bilinear: options for straight line or great circle – Added for MetOffice customers because of “accuracy” questions Normalization options for conservative: destination area or fraction – Added for MetOffice customers Pole options for global spherical logically rectangular grids: – Full circle average, N-point average, teeth, no pole Other: masking, user area, ignore unmapped,…

Regrid: Spherical Grids Support grids with spherical (lon, lat, rad) coordinates Mix and match pairs of: – Global 2D logically rectangular Grids – Regional 2D logically rectangular Grids – 2D unstructured Meshes composed of polygons with any number of sides: ESMF internally represents these as triangles and quadrilaterals Supported elements: triangles, quadrilaterals, pentagons, hexagons,… – Multi-patch grids (e.g. cubed spheres) currently supported via Meshes HOMME Cubed Sphere Grid with Pentagons Courtesy Mark Taylor of Sandia FIM Unstructured Grid Regional Grid

Regrid: Cartesian Grids In addition, regridding supports Cartesian (x,y,z) coordinates: – Regridding between any pair of: 2D Meshes composed of polygons with any number of sides 2D logically rectangular Grids composed of a single patch – Bilinear or Conservative regridding between any pair of: 3D Meshes composed of hexahedrons 3D logically rectangular Grids composed of a single patch 2D Unstructured Mesh From 3D Grid3D Unstructured Mesh

Regrid Weight Calculation Performance Platform: IBM IDataPlex cluster ( Yellowstone at NCAR)

Outline ESMF Overview ESMF Infrastructure ESMF Superstructure New Directions: NUOPC/ESPS and Cupid Current Status and Future Work

ESMF Superstructure State: structure for transferring data between models in a standard way. Can contain Array, Field, Bundles, other States, etc… Gridded Component: wraps a model and allows it to be called in a standard way. Coupler Component: wraps user code for translating data between models and allows it to be called in a standard way. Coupler Comp. Grid Comp. State

Component Hierarchy Using superstructure components can be arranged hierarchically, helping to organize complex models Different groups may create different kinds or levels of components ESMF components in the GEOS-5 atmospheric GCM

Component Overhead Overhead of ESMF component wrapper around native CCSM4 component. No significant performance overhead (<3% is typical) Platform: IBM Power 575, bluefire, at NCAR

Outline ESMF Overview ESMF Infrastructure ESMF Superstructure New Directions: NUOPC/ESPS and Cupid Current Status and Future Work

The initial ESMF software fell short of the vision for common infrastructure in several ways: 1.Implementations of ESMF could vary widely and did not guarantee a minimum level of technical interoperability among sites - creation of the NUOPC Layer 2.It was difficult to track who was using ESMF and how they were using it – initiation of the Earth System Prediction Suite 3.There was a significant learning curve for implementing ESMF in a modeling code – Cupid Integrated Development Environment New development directions address these gaps… New Directions

NUOPC Layer 1. Implementations of ESMF could vary widely and did not guarantee a minimum level of technical interoperability among sites National Unified Operational Prediction Capability: Consortium of U.S. operational weather and water prediction centers. Participants: NOAA, Navy, Air Force, NASA, and other associated modeling groups. Overall goals: – Improve collaboration among agencies. – Accelerate the transition of new technology into the operational centers. Technical goal: Increase interoperability of ESMF-based applications NUOPC websites: – –

The NUOPC Layer An interoperability layer on top of ESMF that adds: Definitions for the component interactions during Initialize, Run, Finalize. Generic components that provide a standard implementation of interoperable components. A field dictionary, based on Climate & Forecast (CF) conventions, as the basis for a standard identification of fields between components. Mechanisms to report component incompatibilities detected during run-time. A compliance checker option that serves as a development and debugging tool. A collection of example applications. – See:

Model : Implements a specific physical domain, e.g. atmosphere, ocean, wave, ice. Mediator : Scientific coupling code (flux calculations, accumulation, averaging, etc.) between (potentially multiple) Models. Connector : Connects pairs of components in one direction, e.g. Model to/from Model, or Model to/from Mediator. Executes simple transforms (Regrid/Redist, units). Driver : Provides a harness for Models, Mediators, and Connectors (supporting hierarchies). Coordinates initialize and run sequences. NUOPC Layer Generic Components

NUOPC Layer Examples

2. It was difficult to track who was using ESMF and how they were using it The Earth System Prediction Suite (ESPS) is a collection of weather and climate modeling codes that use ESMF with the NUOPC conventions. The ESPS makes clear which codes are available as ESMF components and modeling systems. ESPS website: Inclusion criteria for NUOPC: ESPS components and coupled modeling systems are NUOPC-compliant. A minimal, prescribed set of model documentation that conforms to the Common Information Model standard is provided for each version of the ESPS component or modeling system. ESPS codes must have clear terms of use (e.g. public domain statement, open source license, proprietary status), and must have a way for credentialed ESPC collaborators to request access. Regression tests are provided for each component and modeling system. There is a commitment to continued NUOPC compliance and ESPS participation for new versions of the code. The Earth System Prediction Suite

Currently, components in the ESPS can be of the following types: coupled system, atmosphere, ocean, wave, sea ice Target codes include: The Community Earth System Model (CESM) and its constituent components The NOAA Environmental Modeling System (NEMS), including the new Climate Forecast System The MOM5 and HYCOM oceans SWAN and WaveWatch 3 wave models The Navy Global Environmental Model (NavGEM)-HYCOM-CICE coupled system The Navy Coupled Ocean Atmosphere Mesoscale Prediction System (COAMPS) and COAMPS Tropical Cyclone (COAMPS-TC) NASA GEOS-5 NASA ModelE Model Codes in the ESPS

Cupid Development and Training Environment 3. There was a significant learning curve for implementing ESMF in a modeling code CUPID GOAL: Make ESMF training and development simpler and more appealing NOAA CIRES, Georgia Institute of Technology, and NASA GISS/GSFC collaboration Eclipse-based “Integrated Development Environment” or IDE Customized for ESMF applications with NUOPC conventions Cupid is a working prototype expected to be ready for first public release in Cupid project: Cupid tutorial: ide/cupid/blob/master/org.earthsystemcurator.cupid.nuopc.fsml/doc/latex/cupid.pdf?raw=true ide/cupid/blob/master/org.earthsystemcurator.cupid.nuopc.fsml/doc/latex/cupid.pdf?raw=true

Cupid Development and Training Environment Select sample code or model Pick a training problem (or coupled model) Generate a framework-aware outline of the source code Navigate around the source code using the outline Use an editor to modify the source code Automatically generate code needed for NUOPC compliance Compile and run locally or on a cloud (currently Amazon Web Services) Source code editor Console for viewing output Project explorer NUOPC outline

Outline ESMF Overview ESMF Infrastructure ESMF Superstructure New Directions: NUOPC/ESPS and Cupid Current Status and Future Work

Current Status Released ESMF 6.3.0r in January Highlights: Added support for n-gons in Mesh Great circle paths for bilinear Released ESMF 6.3.0rp1 in July Highlights: Python interface (ESMPy) brought into ESMF source Fraction normalization for conservative – Allows regrid of partial destination cells without user normalization

Scheduled for Upcoming Releases Support for 4-sided concave cells in regridding – now all cases work correctly (7.0.0) – Implemented and available as a snapshot For ESMPy, removed requirement that cell centers of Fields be defined, even for operations where they were not needed (7.0.0) – Implemented and available as a snapshot Higher order conservative regridding (7.0.0) Breaking up grid files to increase maximum grid size possible for interpolation weight generation (7.0.0) MOAB finite element library integration: (7.0.0 & 7.1.0) – Introducing MOAB finite element library in addition to ESMF native finite element library – Will be testing to see if we replace native library with MOAB – Would bring in support for higher order elements Extrapolation of points that lie outside the source grid (7.0.0)

References Patch interpolation: 1.Khoei S.A., Gharehbaghi A. R. The superconvergent patch recovery technique and data transfer operators in 3d plasticity problems. Finite Elements in Analysis and Design, 43(8), Hung K.C, Gu H., Zong Z. A modified superconvergent patch recovery method and its application to large deformation problems. Finite Elements in Analysis and Design, 40(5-6), If you have questions or requests, come talk to me, or