1. Mariana Vertenstein (CGD)
Facilitating coupled model development for both climate and weather via CIME (Common Infrastructure for Modeling the Earth)
Mariana Vertenstein (CGD)

2. Outline What is a big challenge for coupled earth system modeling?
What is CIME and what is it comprised of?
How does CIME address challenges of coupled earth system modeling?
Flexible coupling infrastructure
User-friendly case-control system
User-friendly regression testing, unit testing, porting and verification utilities
What are next steps for CIME to address forecast needs?
How are different parts of CIME collaboratively developed and prioritized?

3. What is a coupled model and associated complexities?
A set of fully independent component models representing different parts of the earth system each at different time scales, potentially different resolutions and different processor layouts
In order to provide dynamic feedbacks, the component models periodically exchange state and flux information via a coupling framework
Managing a coupled model requires managing complexity of configuring, building and running a variety of component models, resolutions and processor layouts
CIME was developed to address these challenges!

4. Why do we need a coupling infrastructure?
Driver/
Mediator
river
land ice
land
sea-ice
wave
atm
ocean
Same grid, same decomposition and processors –each component has its own data structures and its own decomposition - Swap out data/prognostic need a general coupling interface
The internals of the components do not know anything about coupling data structures.
An application program interface layer (in green) permits components to translate their data to coupler data structures and vice versa. This API is often referred to as an coupling cap.
The API is the key thing that enables modularity – the driver mediator has no idea what type of components is being used – and therefore data and prognostic components can be swapped easily at compile time.
This is a key feature feature that enables feedbacks to be activated and deactivated.
Components do not exchange data with each other – but only via communication with a mediator
Each component can run on its own grid, decomposition, data structures and processor layout
Driver runs on all Pes and controls the temporal evolution of the system
Mediator does remapping and merging from source -> destination comps
Currently Coupling framework is MCT – moving to NUOPC/ESMF

5. Statistical Verification
CIME is comprised of 3 categories – case control system, utilities, workflow
Case Control System Creating/Building/Running Experiments System Testing Unit Testing
Statistical Verification
Tool (PyCECT)
Mapping weight generation and checking tools (ESMF_Regrid and runoff map maker)
Utilities in CIME repository
CIME Compliant
Workflow
Coupling
Infrastructure
Cylc Workflow
Post Processing/
Diagnostics
Because its an open system, people
Data Models
Externals
PIO
Timers
MCT
Core CIME Functionality

6. CIME Coupling Infrastructure

7. Controlling Component Feedbacks
Active ocean
POP/MOM
Active atmosphere
CAM
CAM and POP/MOM effect each
other during simulation
Active atmosphere CAM
Data OCN
Only Data OCN effects CAM – prescribed SST
Coupling framework
enables feedbacks to
be easily turned off and
on for any component
KEY to adding to new
science to a component
Why are feedbacks important? If trying to improve CAM, then need to make sure that only change atmosphere and not how ocean is responding to atmosphere. Ocean stays fixed and can study just want is going on in the atmosphere.

8. Inter-component Activation and Deactivation of Feedbacks
Driver/
Mediator
Data
ROF
Data ICE
MOSART
CISM
CICE
WW3
CTSM
Data LND
Data WAV
POP
MOM6
SOM/DOM
CAM/MPAS
Data ATM
The 7 CESM components exchange data via communication with a driver/mediator that is based on the MCT (modeling coupling toolkit)
The internals of the components do not know anything about MCT data structures.
An application program interface layer (in green) permits components to translate their data to MCT data and vice versa.
This API is often referred to as an MCT coupling cap.
The API is the key thing that enables modularity – the driver mediator has no idea what type of components is being used – and therefore data and prognostic components can be swapped easily at compile time.
This is a key feature feature that enables feedbacks to be activated and deactivated.
Coupling infrastructure has permits interoperability of active and data components
Inter-component feedbacks are easily activated and deactivated using CIME case control system

9. “stand-alone” CAM Configuration
Driver/
Mediator
CTSM
CICE
Prescribed mode
Data Ocn
Slab Ocn or
Prescribed SST
The 7 CESM components exchange data via communication with a driver/mediator that is based on the MCT (modeling coupling toolkit)
The internals of the components do not know anything about MCT data structures.
An application program interface layer (in green) permits components to translate their data to MCT data and vice versa.
This API is often referred to as an MCT coupling cap.
The API is the key thing that enables modularity – the driver mediator has no idea what type of components is being used – and therefore data and prognostic components can be swapped easily at compile time.
This is a key feature feature that enables feedbacks to be activated and deactivated.
All components are on same grid – but each can have its own decomposition that ensures optimal load balancing
CTSM, CICE and DOCN can all run concurrently on disjoint PES, optimizing performance

10. CIME Case Control System

11. CIME Case Control System
Complexity in CESM continues to grow rapidly
currently support hundreds of out-of-the box experimental configurations and tests. Creating a new experiment is very complex – no single user can know the entire details of all components!
CIME Case Control System is a python based object-oriented package
Lets users easily create customized experiments (using only 4 commands) with desired target feedbacks, component resolutions and component configurations.
Keeps complexity hidden from users – not dependent on the users having expert knowledge of the entire system.
Parallel diagnostic packages can be run from case directory
Provenance: experiment can be saved into run database so that it can be duplicated at a later time
Can easily be plugged into CYCL workflow
You can run the active land with and without a carbon/nitrogen cycle. You can run the atmosphere with and without atmospheric chemistry. You can run the ocean with and without biogeochemistry component.

12. CIME Regression and Unit Testing
Case Control System is accompanied by an extensible and robust regression testing framework that is utilized by all CESM components
All prognostic components leverage the testing framework – component specific tests are specified by component itself - not centralized
Can run hundreds of tests with different resolutions, feedbacks, science complexity, with one command
You can run the active land with and without a carbon/nitrogen cycle. You can run the atmosphere with and without atmospheric chemistry. You can run the ocean with and without biogeochemistry component.

13. Next Steps in CIME Development for Forecasting Needs

14. New Features planned for CIME
Addition of new capabilities in the case control system to accommodate forecast capability and not just climate capability
Migration of coupling framework to ESMF/NUOPC
Will bring in run-time regridding and run-time sequencing of components
ESMF/NUOPC is becoming the de-facto coupling standard – this will lead to enhanced interoperability of the system
Working jointly with NOAA to accomplish this
Coupling infrastructure will be that used by EMC/UFS

15. Thank you Questions?

16. CIME Governance and Development

17. How is CIME developed?
Open joint collaboration on public GitHub repository between DOE/NCAR software developers without any formal governance
CIME is developed and tested stand-alone
CIME stand-alone continuous integration (Travis CI)
CIME stand-alone regression tests and unit tests (required for pull requests)
Prognostic component regression tests use CIME testing framework as well

18. Why does this work (1)
CIME infrastructure has been set up to permit this joint open collaboration
Modular object-oriented python (and fortran) code reuse between efforts
Shared code whenever possible
Separation when necessary
machines, compilers have are implemented in separate locations in CIME – but use the same underlying code

19. Why does this work (2) GitHub is a transformative way to collaborate
Issues are raised in GitHub and resolved collaboratively
Pull requests (PRs) are reviewed by both groups and the implementations must satisfy both groups in order for the PR to be accepted
In practice, GitHub this process easy and transparent
Documentation has been created via sphinx/rst and has been jointly developed and updated

20. CIME Tool for Verification and Porting

21. CESM Ensemble Consistency Test (ECT)
Changes to hardware
or software environment
or CESM code
CESM
Data ≠
Big question: If , is the new result correct?
Alternative question:
Is statistically distinguishable from ?

22. CESM Ensemble Consistency Test (ECT)
Create accepted ensemble
Statistically quantify variability
Create "new" runs
Evaluate: PASS or FAIL
Approach: Evaluate new data in the context of an “accepted” ensemble of CESM runs
CESM-software
engineers
CESM-user
Our new approach is that the climate system model has some natural variability that we are going to represent quantitatively using an ensemble of runs and use that as our baseline for accepted variability
Complete software suite for end users, computationally very efficient
Bit-for-bit is too restrictive – need to be able to optimize code, new hardware technologies, more aggressive compiler options
Easy, automated: the end user does not need to do much of this
climate-modeling expertise is not required!
option when bit-for-bit reproducibility is not possible:
new algorithms, solvers, compiler options, hardware technologies
Highlights:
enables “letting go” of bit-for-bit reproducibility
objective, user-friendly 9 time-step test
rapid feedback for model developers