Mariana Vertenstein (CGD) Facilitating coupled model development for both climate and weather via CIME (Common Infrastructure for Modeling the Earth) Mariana Vertenstein (CGD)

Outline What is a big challenge for coupled earth system modeling? What is CIME and what is it comprised of? How does CIME address challenges of coupled earth system modeling? Flexible coupling infrastructure User-friendly case-control system User-friendly regression testing, unit testing, porting and verification utilities What are next steps for CIME to address forecast needs? How are different parts of CIME collaboratively developed and prioritized?

What is a coupled model and associated complexities? A set of fully independent component models representing different parts of the earth system each at different time scales, potentially different resolutions and different processor layouts In order to provide dynamic feedbacks, the component models periodically exchange state and flux information via a coupling framework Managing a coupled model requires managing complexity of configuring, building and running a variety of component models, resolutions and processor layouts CIME was developed to address these challenges!

Why do we need a coupling infrastructure? Driver/ Mediator river land ice land sea-ice wave atm ocean Same grid, same decomposition and processors –each component has its own data structures and its own decomposition - Swap out data/prognostic need a general coupling interface The internals of the components do not know anything about coupling data structures. An application program interface layer (in green) permits components to translate their data to coupler data structures and vice versa. This API is often referred to as an coupling cap. The API is the key thing that enables modularity – the driver mediator has no idea what type of components is being used – and therefore data and prognostic components can be swapped easily at compile time. This is a key feature feature that enables feedbacks to be activated and deactivated. Components do not exchange data with each other – but only via communication with a mediator Each component can run on its own grid, decomposition, data structures and processor layout Driver runs on all Pes and controls the temporal evolution of the system Mediator does remapping and merging from source -> destination comps Currently Coupling framework is MCT – moving to NUOPC/ESMF

Statistical Verification CIME is comprised of 3 categories – case control system, utilities, workflow Case Control System Creating/Building/Running Experiments System Testing Unit Testing Statistical Verification Tool (PyCECT) Mapping weight generation and checking tools (ESMF_Regrid and runoff map maker) Utilities in CIME repository CIME Compliant Workflow Coupling Infrastructure Cylc Workflow Post Processing/ Diagnostics Because its an open system, people Data Models Externals PIO Timers MCT Core CIME Functionality

CIME Coupling Infrastructure

Controlling Component Feedbacks Active ocean POP/MOM Active atmosphere CAM CAM and POP/MOM effect each other during simulation Active atmosphere CAM Data OCN Only Data OCN effects CAM – prescribed SST Coupling framework enables feedbacks to be easily turned off and on for any component KEY to adding to new science to a component Why are feedbacks important? If trying to improve CAM, then need to make sure that only change atmosphere and not how ocean is responding to atmosphere. Ocean stays fixed and can study just want is going on in the atmosphere.

Inter-component Activation and Deactivation of Feedbacks Driver/ Mediator Data ROF Data ICE MOSART CISM CICE WW3 CTSM Data LND Data WAV POP MOM6 SOM/DOM CAM/MPAS Data ATM The 7 CESM components exchange data via communication with a driver/mediator that is based on the MCT (modeling coupling toolkit) The internals of the components do not know anything about MCT data structures. An application program interface layer (in green) permits components to translate their data to MCT data and vice versa. This API is often referred to as an MCT coupling cap. The API is the key thing that enables modularity – the driver mediator has no idea what type of components is being used – and therefore data and prognostic components can be swapped easily at compile time. This is a key feature feature that enables feedbacks to be activated and deactivated. Coupling infrastructure has permits interoperability of active and data components Inter-component feedbacks are easily activated and deactivated using CIME case control system

“stand-alone” CAM Configuration Driver/ Mediator CTSM CICE Prescribed mode Data Ocn Slab Ocn or Prescribed SST The 7 CESM components exchange data via communication with a driver/mediator that is based on the MCT (modeling coupling toolkit) The internals of the components do not know anything about MCT data structures. An application program interface layer (in green) permits components to translate their data to MCT data and vice versa. This API is often referred to as an MCT coupling cap. The API is the key thing that enables modularity – the driver mediator has no idea what type of components is being used – and therefore data and prognostic components can be swapped easily at compile time. This is a key feature feature that enables feedbacks to be activated and deactivated. All components are on same grid – but each can have its own decomposition that ensures optimal load balancing CTSM, CICE and DOCN can all run concurrently on disjoint PES, optimizing performance

CIME Case Control System

CIME Case Control System Complexity in CESM continues to grow rapidly currently support hundreds of out-of-the box experimental configurations and tests. Creating a new experiment is very complex – no single user can know the entire details of all components! CIME Case Control System is a python based object-oriented package Lets users easily create customized experiments (using only 4 commands) with desired target feedbacks, component resolutions and component configurations. Keeps complexity hidden from users – not dependent on the users having expert knowledge of the entire system. Parallel diagnostic packages can be run from case directory Provenance: experiment can be saved into run database so that it can be duplicated at a later time Can easily be plugged into CYCL workflow You can run the active land with and without a carbon/nitrogen cycle. You can run the atmosphere with and without atmospheric chemistry. You can run the ocean with and without biogeochemistry component.

CIME Regression and Unit Testing Case Control System is accompanied by an extensible and robust regression testing framework that is utilized by all CESM components All prognostic components leverage the testing framework – component specific tests are specified by component itself - not centralized Can run hundreds of tests with different resolutions, feedbacks, science complexity, with one command You can run the active land with and without a carbon/nitrogen cycle. You can run the atmosphere with and without atmospheric chemistry. You can run the ocean with and without biogeochemistry component.

Next Steps in CIME Development for Forecasting Needs

New Features planned for CIME Addition of new capabilities in the case control system to accommodate forecast capability and not just climate capability Migration of coupling framework to ESMF/NUOPC Will bring in run-time regridding and run-time sequencing of components ESMF/NUOPC is becoming the de-facto coupling standard – this will lead to enhanced interoperability of the system Working jointly with NOAA to accomplish this Coupling infrastructure will be that used by EMC/UFS

Thank you Questions?

CIME Governance and Development

How is CIME developed? Open joint collaboration on public GitHub repository between DOE/NCAR software developers without any formal governance CIME is developed and tested stand-alone CIME stand-alone continuous integration (Travis CI) CIME stand-alone regression tests and unit tests (required for pull requests) Prognostic component regression tests use CIME testing framework as well

Why does this work (1) CIME infrastructure has been set up to permit this joint open collaboration Modular object-oriented python (and fortran) code reuse between efforts Shared code whenever possible Separation when necessary machines, compilers have are implemented in separate locations in CIME – but use the same underlying code

Why does this work (2) GitHub is a transformative way to collaborate Issues are raised in GitHub and resolved collaboratively Pull requests (PRs) are reviewed by both groups and the implementations must satisfy both groups in order for the PR to be accepted In practice, GitHub this process easy and transparent Documentation has been created via sphinx/rst and has been jointly developed and updated

CIME Tool for Verification and Porting

CESM Ensemble Consistency Test (ECT) Changes to hardware or software environment or CESM code CESM Data ≠ Big question: If , is the new result correct? Alternative question: Is statistically distinguishable from ?

CESM Ensemble Consistency Test (ECT) Create accepted ensemble Statistically quantify variability Create "new" runs Evaluate: PASS or FAIL Approach: Evaluate new data in the context of an “accepted” ensemble of CESM runs CESM-software engineers CESM-user Our new approach is that the climate system model has some natural variability that we are going to represent quantitatively using an ensemble of runs and use that as our baseline for accepted variability Complete software suite for end users, computationally very efficient Bit-for-bit is too restrictive – need to be able to optimize code, new hardware technologies, more aggressive compiler options Easy, automated: the end user does not need to do much of this climate-modeling expertise is not required! option when bit-for-bit reproducibility is not possible: new algorithms, solvers, compiler options, hardware technologies Highlights: enables “letting go” of bit-for-bit reproducibility objective, user-friendly 9 time-step test rapid feedback for model developers