1. ESMF Regridding Update
Robert Oehmke, Peggy Li, Ryan O’Kuinghttons, Mat Rothstein, Joseph Jacob
NOAA Cooperative Institute for Research in Environmental Sciences
University of Colorado, Boulder
3rd Workshop on Coupling Technologies for Earth System Models
April 21, 2015

2. Context
The Earth System Modeling Framework (ESMF) is software for building and coupling weather, climate, and related models.
This talk is about the regridding part of ESMF
The work presented here is finished and we expect it to come out in ESMF due late spring/early summer
Also available before that as a development snapshot

3. ESMF Regridding
Regridding (or remapping or interpolation) is the process of moving data from one grid to another while preserving qualities of the original data.
ESMF regridding is:
Flexible:
Computes weights between a wide range of grids: structured and unstructured, global and regional, 2D and 3D, spherical and Cartesian
Options for interpolation method, pole treatment, masked points, …
Well Tested and Portable:
>200 regridding cases tested every night
>40 different OS/Compiler/MPI combinations tested every night
Parallel and Fast:
Able to compute weights in minutes which before took hours
Able to compute weights between very large grids
Community developed:
Supported by NASA, NOAA, DOD and NSF funding
Well established (since 2005) community processes for prioritization, support and review.
Development priorities set by users through quarterly Change Review Board (CRB) meetings

4. Interfaces Complete F95 API: Limited C API: Limited Python API:
use ESMF
Derived types and methods
Investigating moving to Fortran 2003
Limited C API:
#include “ESMC.h”
Structs and methods
Limited Python API:
Import ESMPy
Classes with methods
Applications:
File-based interpolation weight generation: mpirun –np <N> ESMF_RegridWeightGen –s ….
(NEW!) File-based weight generation AND application of weights: mpirun –np <N> ESMF_Regrid –s…
! Create Grids
srcGrid=ESMF_GridCreate(…)
dstGrid=ESMF_GridCreate(…)
! Create Fields
srcField=ESMF_FieldCreate(srcGrid,…)
dstField=ESMF_FieldCreate(dstGrid,…)
! Calc regrid sparse matrix (routeHandle)
ESMF_FieldRegridStore(srcField, dstField, routeHandle,…)
do i=1,…
! Compute new srcField ….
! Apply regrid sparse matrix (routeHandle)
ESMF_FieldRegrid(srcField, dstField,
routeHandle)
enddo

5. Supported Geometry Types
Grid:
Structured representation of a region
A logically rectangular tile
Mesh:
Unstructured representation of a region
In 2D: polygons with any number of sides
including concave
In 3D: tetrahedrons & hexahedrons
LocStream (Location Stream):
NEW!
Set of disconnected points
E.g. locations of observations
Very flexible and efficient
Can’t be used with every regrid method
+ Most structured to least structured
+ The most structured is what we call a Grid. This is a logically rectangular tile. Note that although memory has a logically rectangular structure the coordinates don’t need to be so can specify curvilinear grids.
+ The next is what we call a Mesh.
+ Can now handle all concave cells. Previously we couldn’t do those with 4 sides.
+ LocStream good. Previously people would have to force sets of points into a Mesh. This is annoying, but also inefficient because then you’re storing more info than you need.

6. Supported Grid File Formats
SCRIP:
Format used by SCRIP regridding tool
2D Spherical
Logically rectangular Grids or unstructured Meshes
ESMF unstructured:
Custom ESMF format
2D or 3D / Spherical or Cartesian
Unstructured Meshes
UGRID:
Proposed CF convention
GRIDSPEC:
CF convention
ESMF currently supports: 2D spherical single tile Grids

7. Regrid Methods Bilinear: Higher order patch recovery:
Destination is a linear combination of source cell corners
Weights based distance from corners
LocStreams can be destination
Higher order patch recovery:
Multiple polynomial patches represent region around source cell
Destination is linear combination of patch values
Yields better derivatives/smoother results than bilinear
Based on “patch recovery” used in finite element modeling [1][2]
Nearest neighbor:
Destination is equal to closest source point (or vise versa)
LocStreams can be source or destination
First order conservative:
Destination is combination of intersecting source cell areas
Preserves integral of data across interpolation

8. Other Options (NEW!) Path between points in bilinear on a sphere:
Straight line
Great circle
Options for extrapolating across pole region:
Full circle average
N-point average
Teeth
No pole
Others:
Masking
(New!) Normalization options for conservative: destination area or fraction
User area
Ignore unmapped, Ignore degenerate
Full circle avg.
N-point avg
Teeth
No Pole
We also support some other options in regrid.
Previously using straight line distance would sometimes give unexpected distirubtion of data inside a coarse src cells given the curve of the sphere. Great circle gives results more like you would expect given the sphere’s shape.

9. Spherical Regrid Support
Regrid works with spherical (lon, lat, radius) coordinates
All regrid methods supported between any pair of:
2D Global or 2D regional logically rectangular Grids
2D Unstructured Meshes composed of polygons with any number of sides
2D Multi-patch grids (e.g. cubed spheres) currently supported via Meshes
(NEW! More Accurate!) Bilinear supported between any pair of:
3D Meshes composed of hexahedrons
3D Global or regional logically rectangular Grids
LocStreams supported for above depending on regrid method
3D Global Spherical Grid
FIM Unstructured Grid
Regional Grid

10. Cartesian Regrid Support
Regrid works with Cartesian (x,y,z) coordinates
All regrid methods between any pair of:
2D Meshes composed of polygons with any number of sides
2D logically rectangular Grids
Bilinear, conservative, or nearest neighbor between any pair of:
3D Meshes composed of hexahedrons
3D logically rectangular Grids
LocStreams supported for above depending on regrid method
2D Unstructured Mesh
From
+ point to pictures
+ Support locstreams any place the method allows
3D Grid
3D Unstructured Mesh

11. Regrid Weight Calculation Performance
30 Million cells to around 4 Million Cells
Pretty fast under 30s
Scales well out to 4096 processors
Platform: IBM IDataPlex cluster (Yellowstone at NCAR)
Grid size: ~30 million cells and ~4 millions cells

12. Other Tools Using ESMF Regrid
Ultrascale Visualization Climate Data Analysis Tool (UV-CDAT):
Package designed for analyzing large climate data sets
Uses ESMF regridding via ESMPy
Recently won Federal Laboratory Consortium technology transfer award
Cf-python:
Python package for manipulating cf data and files
NCAR Command Language (NCL):
Language for scientific data analysis and visualization
Uses ESMF regridding via ESMF_RegridWeightGen application

13. Selected Users NOAA Environmental Modeling System (NEMS):
Under development
Aimed to be the backbone of next generation forecast systems
ESMF regridding used to transfer data between models
Community Earth System Model (CESM):
Has used ESMF regridding for 5+ years
Currently uses ESMF_RegridWeightGen, but are working towards online regridding
Met Office:
Multiple users of ESMF regridding there
Have been adding functionality for them (e.g. concave, great circle,…)
NOAA Space Weather Prediction Center (SWPC):
Working towards coupled 3D ionosphere model using ESMF
New 3D spherical bilinear reduces error by 40x over Cartesian

14. Scheduled for Upcoming Releases
Higher-order conservative regridding (7.1.0)
Breaking up grid files to increase maximum grid size possible for interpolation weight generation (7.1.0)
Extrapolation of points that lie outside the source grid (7.1.0)
Dynamic masking during sparse matrix multiply (7.1.0)
Additional weight file formats for ESMF_RegridWeightGen (7.1.0)
Other SCRIP format
Slim version
Representation of higher-order elements (7.2.0)

15. If you have questions or requests,
References
Patch interpolation:
Khoei S.A., Gharehbaghi A. R. The superconvergent patch recovery technique and data transfer operators in 3d plasticity problems. Finite Elements in Analysis and Design, 43(8), 2007.
Hung K.C, Gu H., Zong Z. A modified superconvergent patch recovery method and its application to large deformation problems. Finite Elements in Analysis and Design, 40(5-6), 2004.
If you have questions or requests,
come talk to me, or
+ Here are some references for the patch interpolation.
+ Any questions?