Community infrastructure for building and coupling high performance climate, weather, and coastal models Cecelia DeLuca NOAA / CIRES University of Colorado, Boulder ESMF website: Building Community Codes for Effective Scientific Research on HPC Platforms September 6-7, 2012

Outline Vision and Context Technical Overview User Community Values, Processes and Governance Future

Vision Earth system models that can be built, assembled and reconfigured easily, using shared toolkits and standard interfaces. A growing pool of Earth system modeling components that, through their broad distribution and ability to interoperate, promotes the rapid transfer of knowledge. Earth system modelers who are able to work more productively, focusing on science rather than technical details. An Earth system modeling community with cost-effective, shared infrastructure development and many new opportunities for scientific collaboration. Accelerated scientific discovery and improved predictive capability through the social and technical influence of ESMF.

Computational Context Teams of specialists, often at different sites, contribute scientific or computational components to an overall modeling system Components may be at multiple levels: individual physical processes (e.g. atmospheric chemistry), physical realms (e.g. atmosphere, ocean), and members of same or multi-model ensembles (e.g. “MIP” experiments) Components contributed from multiple teams must be coupled together, often requiring transformations of data in the process(e.g. grid remapping and interpolation, merging, redistribution) Transformations are most frequently 2D data, but 3D is becoming more common There is an increasing need for cross-disciplinary and inter-framework coupling for climate impacts Running on tens of thousands of processors is fairly routine; utilizing hundreds of thousands of processors or GPUs is less common Modelers will tolerate virtually no framework overhead and seek fault tolerance and bit reproducibility Provenance collection is increasingly important for climate simulations

Architecture Low Level Utilities Fields and Grids Layer Model Layer Components Layer Gridded Components Coupler Components ESMF Infrastructure User Code ESMF Superstructure MPI, NetCDF, … External Libraries The Earth System Modeling Framework (ESMF) provides a component architecture or superstructure for assembling geophysical components into applications. ESMF provides an infrastructure that modelers use to – Generate and apply interpolation weights – Handle metadata, time management, I/O and communications, and other common functions The ESMF distribution does not include scientific models

Components ESMF is based on the idea of components – sections of code that are wrapped in standard interfaces Components can be arranged hierarchically, helping to organize the structure of complex models Different modeling groups may create different kinds or levels of components Some of the ESMF components in the GEOS-5 atmospheric GCM

Applications of information layer ESMF as an Information Layer Native model data structures Standard data structures Standard metadata Parallel generation and application of interpolation weights Run-time compliance checking of metadata and time behavior Fast parallel I/O Redistribution and other parallel communications Automated documentation of models and simulations Ability to run components in workflows and as web services Field Grid Clock Component Attributes: CF conventions, ISO standards, METAFOR Common Information Model Structured model information stored in ESMF wrappers User data is referenced or copied into ESMF structures modules fields grids timekeeping ESMF data structures

Standard Interfaces All ESMF components have the same three standard methods: – Initialize – Run – Finalize Each standard method has the same simple interface: call ESMF_GridCompRun (myComp, importState, exportState, clock, …) Where: myComp points to the component importState is a structure containing input fields exportState is a structure containing output fields clock contains timestepping information Steps to adopting ESMF Divide the application into components (without ESMF) Copy or reference component input and output data into ESMF data structures Register components with ESMF Set up ESMF couplers for data exchange Interfaces are wrappers and can often be set up in a non-intrusive way

Data Structures ESMF represents data as Fields that can be built on four discretization types: – Logically rectangular grids, which may be connected at the edges – Unstructured meshes – Observational data streams – Exchange grids – grids that are the union of the grids of components being coupled, and reference the data on the original grids ESMF can transform data among these representations, using an underlying finite element unstructured mesh engine

Component Overhead Representation of the overhead for ESMF wrapped native CCSM4 component For this example, ESMF wrapping required NO code changes to scientific modules No significant performance overhead (< 3% is typical) Few code changes for codes that are modular Platform: IBM Power 575, bluefire, at NCAR Model: Community Climate System Model (CCSM) Versions: CCSM_4_0_0_beta42 and ESMF_5_0_0_beta_snapshot_01 Resolution: 1.25 degree x 0.9 degree global grid with 17 vertical levels for both the atmospheric and land model, i.e. 288x192x17 grid. The data resolution for the ocean model is 320x384x60.

ESMF Regridding ESMF offers extremely fast parallel regridding with many options (regional, global; bilinear, higher order, first order conservative methods; logically rectangular grids and unstructured meshes; pole options; 2D or 3D; invocation as offline application or during model run) Summary of features: blic/ESMF_5_3_0/esmf_5_3_0_regridding_status.htm HOMME Cubed Sphere Grid with Pentagons Courtesy Mark Taylor of Sandia FIM Unstructured Grid Regional Grid ESMF supported grids IMPACT: “use of the parallel ESMF offline regridding capability has reduced the time it takes to create CLM surface datasets from hours to minutes” - Mariana Vertenstein, NCAR IMPACT: “use of the parallel ESMF offline regridding capability has reduced the time it takes to create CLM surface datasets from hours to minutes” - Mariana Vertenstein, NCAR

ESMF Offline RegridWeightGen Application: – Separate application that is built as part of ESMF, can be used independently – Application generates a netCDF weight file from two netCDF grid files – Supports SCRIP, GRIDSPEC, UGRID, and custom ESMF unstructured format mpirun –np 32 ESMF_RegridWeightGen –s src_grid.nc –d dst_grid.nc –m bilinear –w weights.nc Regridding during model execution: – ESMF library subroutine calls which do interpolation during model run – Can get weights or feed directly into ESMF parallel sparse matrix multiply – Can be used without ESMF components call ESMF_FieldRegridStore(srcField=src, dstField=dst, regridMethod=ESMF_REGRID_METHOD_BILINEAR, routehandle=rh) call ESMF_FieldRegrid(srcField=src, dstField=dst, routehandle=rh) Regridding Modes

9/7/2015 ESMP – Python Interface ● Flexible way to use ESMF parallel regridding ● Separate download: ● Requirements: – Python – Numpy – Ctypes ● Limited platform support: Linux/Darwin, GCC(g++/gfortran), OpenMPI ● Data type: ESMP_Field ● Grid types: – Single-tile 2D logically rectangular type: ESMP_Grid – Unstructured type: ESMP_Mesh ● Support for all ESMF interpolation options

ESMF Web Services for Climate Impacts GOAL: Develop a two-way coupled, distributed, service-based modeling system comprised of an atmospheric climate model and a hydrological model, utilizing standard component interfaces from each domain. IDEA: Bring the climate model to local applications Two way technical coupling completed during 2012 using CAM (with active land) and SWAT (Soil Water Assessment Tool) Utilized switch in CAM’s ESMF component wrapper that enables web service interface Coupled system is run using the OpenMI configuration editor, a web service driver developed in the hydrological community Next steps: using WRF within the CESM framework and an updated hydrological model

Summary of Features Components with multiple coupling and execution modes for flexibility, including a web service execution mode Fast parallel remapping with many features Core methods are scalable to tens of thousands of processors Supports hybrid (threaded/distributed) programming for optimal performance on many computer architectures; works with codes that use OpenMP and OpenACC Time management utility with many calendars, forward/reverse time operations, alarms, and other features Metadata utility that enables comprehensive, standard metadata to be written out in standard formats Runs on 30+ platform/compiler combinations, exhaustive nightly regression test suite (4500+ tests) and documentation Couples Fortran or C-based model components Open source license

Major Users ESMF Components: NOAA National Weather Service operational weather models (GFS, Global Ensemble, NEMS) NASA atmospheric general circulation model GEOS-5 Navy and related atmospheric, ocean and coastal research and operational models – COAMPS, NOGAPS, HYCOM, WaveWatch, others Hydrological modelers at Delft hydraulics, space weather modelers at NCAR and NOAA ESMF Regridding and Python Libraries NCAR/DOE Community Earth System Model (CESM) Analysis and visualization packages: NCAR Command Language, Ultrascale Visualization - Climate Data Analysis Tools (UV-CDAT), PyFerret users Community Surface Dynamics Modeling System

Usage Metrics Updated ESMF component listing at: Includes 85 components with ESMF interfaces, 12 coupled cross- agency modeling systems in space weather, climate, weather, hydrology, and coastal prediction, for operational and research use About 4500 registered downloads

Values and Principles Community driven development and community ownership Openness of project processes, management, code and information Correctness Commitment to a globally distributed and diverse development and customer base Simplicity Efficiency User engagement Environmental stewardship Web link for detail:

Agile Development vs Community Values In general, Agile processes promote to community development. However, there are areas of potential conflict: Agile emphasis on co-location: “most agile methodologies or approaches assume that the team is located in a single team room” (from Miller, Dist. Agile Development at Microsoft, 2008) vs ESMF emphasis on distributed co-development Agile emphasis on single product owner vs ESMF emphasis on multi-party ownership

Making Distributed Co-Development Work Hinges on asynchronous, all-to-all communication patterns: everybody must have information Archived list where all development correspondence gets cc’d Minutes for all telecons Web browsable repositories (main and contributions), mail summary on check-ins Daily, publicly archived test results Monthly archived metrics Public archived trackers (bugs, feature requests, support requests, etc.) Discouraged: IMing, one-to-one correspondence or calls – the medium matters

Change Review Board CRB established as a vehicle for shared ownership through user task prioritization and release content decisions Consists of technical leads from key user communities Not led by the development team! Sets the schedule and expectations for future functionality enhancements in ESMF internal and public distributions – Based on broad user community and stakeholder input – Constrained by available developer resources – Updated quarterly to reflect current realities CRB reviews releases after delivery for adherence to release plan

Governance Highlights Management of ESMF required governance that recognized social and cultural factors as well as technical factors Main practical objectives of governance: Enabling stakeholders to fight and criticize in a civilized, contained, constructive way Enabling people to make priority decisions based on resource realities Observations: Sometimes just getting everyone equally dissatisfied and ready to move on is a victory Thorough, informed criticism is the most useful input a project can get Governance changes and evolves over the life span of a project

Governance Functions Prioritize development tasks in a manner acceptable to major stakeholders and the broader community, and define development schedules based on realistic assessments of resource constraints (CRB) Deliver a product that meets the needs of critical applications, including adequate and correct functionality, satisfactory performance and memory use,... (Core) Support users via prompt responses to questions, training classes, minimal code changes for adoption, thorough documentation,... (Core) Encourage community participation in design and implementation decisions frequently throughout the development cycle (JST) Leverage contributions of software from the community when possible (JST) Create frank and constructive mechanisms for feedback (Adv. Board) Enable stakeholders to modify the organizational structure as required (Exec. Board) Coordinate and communicate at many levels in order to create a knowledgeable and supportive network that includes developers, technical management, institutional management, and program management (IAWG and other bodies)

Governance Executive Board Strategic Direction Organizational Changes Board Appointments Interagency Working Group Stakeholder Liaison Programmatic Assessment & Feedback Advisory Board External Projects Coordination General Guidance & Evaluation Functionality Change Requests Joint Specification Team Requirements Definition Design and Code Reviews External Code Contributions Implementation Schedule Resource Constraints Collaborative Design Beta Testing Working Project Executive Management Reporting weekly Core Development Team Project Management Software Development Testing & Maintenance Distribution & User Support daily annually Change Review Board Development Priorities Release Review & Approval quarterly

Evolution Phase 1: NASA’s Earth Science Technology Office ran a solicitation to develop an Earth System Modeling Framework (ESMF). A multi-agency collaboration (NASA/NSF/DOE/NOAA) won the award. The core development team was located at NCAR. A prototype ESMF software package (version 2r) demonstrated feasibility. Phase 2: New sponsors included Department of Defense and NOAA. A multi-agency governance plan including the CRB was created: Many new applications and requirements were brought into the project, motivating a complete redesign of framework data structures (version 3r). Phase 3: (and beyond) The core development team moved to NOAA/CIRES for closer alignment with federal models. Basic framework development completed with version 5r (ports, bugs, feature requests, user support etc. still require resources). The focus is on increasing adoption and creating a community of interoperable codes.

Technical Evolution: National Unified Operational Prediction Capability (NUOPC) ESMF allows for many levels of components, types of components, and types of connections In order to achieve greater interoperability, usage and content conventions and component templates are needed A tri-agency collaboration (NOAA, Navy, Air Force) is building a “NUOPC Layer” that constrains how ESMF is used, and introduces metadata and other content standards, along with inheritable templates for different usage scenarios A production version of the NUOPC Layer is scheduled for delivery at the end of 2012

CoG Collaboration Environment Right side: Services Central navigation: Templated content Central Section: Freeform content Left side: Auto generated navigation of project freeform content The CoG environment exposes and collates the information needed for distributed, multi-project development, including project repositories, trackers, and governance processes. It does this in an environment that’s linked to data search, metadata, and visualization services and set up to enable component comparisons.

DCMIP on CoG Atmospheric Dynamical Core Model Intercomparison Project Planned MIPs: 2013 Downscaling 2014 Atm-surface hydrology 2014 Frameworks

9/7/2015 Future Governance and process largely follow established patterns Active development ● Python API redesign and addition of features ● Preparation for NUOPC Layer major release ● Web services for climate impacts ● Regridding options – higher order conservative, filling in gaps ● Advanced fault-tolerance ● Performance optimizations and GPUs ● CoG collaboration environment ESMF and related development is supported by NASA, NOAA, NSF, and the Department of Defense.