1. Coupling Climate and Hydrological Models Interoperability Through Web Services Kathy Saint/SGI – NESII Jon Goodall/University of South Carolina Richard Rood/University of Michigan Haihang You/NSF XSEDE Cecelia DeLuca/NOAA NESII CW2013 February 20, 2013

2. Outline Project Objective Motivation System Description
Components
Frameworks
System Driver
Logical Workflow
Data Flow
Architecture
Future Directions

3. Project Objectives
The development of a distributed, service-based modeling system that incorporates both high performance computing and local, PC-based computing resources.
Demonstration of a prototype two-way coupled system utilizing a high performance, highly parallel atmospheric model and a PC-based hydrology code used by local resource managers.
Preservation of native infrastructure for both of the communities involved (climate research, water resources information delivery)

4. Motivation
Hydrological impact studies can be improved when forced with data from climate models [Zeng et al., 2003; Yong et al., 2009] ; regional hydrology can affect climate.
A technology gap exists:
Many hydrological models, impacts models, and end-user services run on personal computers
Most climate models run on high performance supercomputers
Leveraging interface standards ESMF and OpenMI can mitigate the communication difficulties between these model types
ESMF contains web services interfaces that can be used to communicate across a distributed network
Both ESMF and OpenMI are widely used within their respective communities

5. High Performance Computer
System Description
Soil and Water Assessment Tool – SWAT (hydrology model) runs on PC
Community Atmosphere Model - CAM (climate model) runs on HPC
Wrappers for both SWAT and CAM provide OpenMI interface to each model
Driver (OpenMI Configuration Editor) uses OpenMI interface to timestep through models via wrappers
SWAT
OpenMI
CAM OpenMI Wrapper
Driver
Personal Computer
High Performance Computer
ESMF Web Services
ESMF CAM
Component
Access to CAM across the network provided by ESMF Web Services
CAM output data streamed to CAM wrapper via ESMF Web Services

6. Components: SWAT
SWAT is a river basin scale model developed to quantify the impact of land management practices in large, complex watersheds
It was chosen for this project because it is widely used, is open source, and runs on a Windows platform

7. Components: CAM
The atmospheric model chosen for this system is the Community Atmospheric Model (CAM5), part of the Community Climate System Model (CESM1.0.3)
It was chosen because it has ESMF component interfaces, it is widely used, and it is open source

8. Frameworks: Earth System Modeling Framework
Is a high-performance, flexible software infrastructure that increases the ease of use, performance portability, interoperability, and reuse of Earth science applications
Provides an architecture for composing complex, coupled modeling systems and includes array-based, multi-dimensional data structures
Has utilities for developing individual models including utilities to make models self-describing
Web services included in the ESMF distribution allow any networked ESMF component to be available as a web service.

9. Frameworks: OpenMI
The OpenMI Software Development Kit (SDK) is a software library that provides a standardized interface that focuses on time dependent data transfer
Primarily designed to work with systems that run simultaneously, but in a single-threaded environment [Gregerson et al., 2007]
The primary data structure in OpenMI is the ExchangeItem, which comes in the form of an InputExchangeItem and an OutputExchangeItem (single point, single timestep)

10. The system driver Controls the application flow
Implemented using OpenMI’s Configuration Editor
Convenient tool for the testing of the OpenMI implementations and model interactions

11. Hardware Architecture
The Atmospheric Model runs on a HPC platform
Access to the HPC Compute Nodes must be through the Login Nodes
Access to the Login Nodes is through the Virtual Server (Web Svcs)
Personal Computer
(Windows)
High Performance Computer
Virtual
Linux
Server
The Client contains the OpenMI and SWAT software, which run on a Windows platform.
Login Nodes
(kraken)
Compute Nodes (kraken)

12. Software Achitecture Client
Configuration Editor is the driver… it is used to link the models and trigger the start of the run.
Hydrological model (SWAT 2005) is a modified version to work with OpenMI
Access to Atmospheric model (CAM) is done through “wrapper” code that accesses ESMF Web Services via an OpenMI interface
Personal Computer (Windows)
OpenMI
Configuration Editor
OpenMI
SWAT 2005
CAM OpenMI Wrapper
… to Web Services

13. Software Architecture Server
HPC Login Nodes
Linux Server (Web Svr)
Tomcat/Axis2
Process
Controller
Registrar
SOAP Svcs
Job
Scheduler
In some HPC systems, access to nodes can be restrictive. In XSEDE, only the Login Nodes can communicate with the Compute Nodes.
Access to/from external systems can be controlled via “gateway” systems using Web Services.
Running applications (such as CAM Component Svc) on Compute Nodes must be handled by a Job Scheduler.
HPC Compute Nodes
Comp
Svc
Comp
Svc
Comp
Svc
CAM
CAM
CAM

14. Creating a Component Svc From an ESMF Component
Make sure ESMF Component implements standard initialize, run and finalize routines
Modify the driver from this:
gcomp = ESMF_GridCompCreate(name=cname, rc=rc)
call ESMF_GridCompSetServices(gcomp, userRoutine=my_register_routine, rc=rc)
call ESMF_GridCompInitialize(gcomp, rc=rc)
call ESMF_GridCompRun(gcomp, rc=rc)
call ESMF_GridCompFinalize(gcomp, rc=rc)
to this:
call ESMF_WebServicesLoop(gcomp, portNum=27060, rc=rc)
Setup and install Process Controller, Registrar, and Web Server applications

15. Logical Workflow One-Way Coupling
Driver
SWAT/OpenMI
ATM/OpenMI Wrapper
ESMF Web Services
ESMF Component
Initialize
Prepare
GetValues
Finish
Dispose
NewClient
RunTimestep
Finalize
GetData
EndClient
ESMF_GridCompInitialize
ESMF_GridCompRun
ESMF_GridCompFinalize
ValueSet

16. Logical Workflow Two-Way Coupling
Driver
SWAT/OpenMI
ATM/OpenMI Wrapper
ESMF Web Services
ESMF Component
Initialize
Prepare
GetValues
Finish
Dispose
NewClient
RunTimestep
Finalize
GetData
EndClient
ESMF_GridCompInitialize
ESMF_GridCompRun
ESMF_GridCompFinalize
Extrapolate
ValueSet

17. Data Flow One-Way Coupling
High Performance
Computer
Personal Computer
GetValues
ESMF Component/CAM
GetDataValues
SWAT/OpenMI
Input
Exchange Item
ESMF State
CAM/OpenMI
Wrapper
Output
Exchange Item
The data is pulled from the CAM Component to SWAT via the wrapper , initiated by the OpenMI GetValues call; this call is made once per timestep.
Data is exchanged between CAM and SWAT using the OpenMI Exchange Item structures that handle the translation from grid to point values

18. Data Flow Two-Way Coupling
ESMF Component/CAM
ESMF Export State
CAM/OpenMI
Wrapper
Output
Exchange Item
Import
SWAT/OpenMI
Input
GetValues
Personal Computer
High Performance
Computer
GetDataValues
SetInputData
ESMF Import State
In two-way coupling, each model pulls the data from the other model using the OpenMI GetValues method. Extrapolation is used on the first timestep to break the deadlock between the two model requests.
OpenMI Input and Output Exchanges items are again used to exchange and translate the data.

19. Model Configurations CAM SWAT Global atmospheric model
Model run: 1 day
Timestep: 1800 sec
Dynamical core: finite volume
Horizontal grid: 10x15
Export data variables:
surface air temperature
precipitation
wind speed
relative humidity
solar radiation
SWAT
Hydrology science information provided by Jon Goodall of University of S. Carolina
Lake Fork Watershed (TX)
Watershed area: km2
Model run: 2 years, 1977 – 1978
Timestep = 1 day
Weather stations:
wea62 (33.03 N, W)
wea43 (33.25 N, W)
Export data variables:
evapotranspiration

20. Scaling Analysis 4 areas of increasing size
3 variations of CAM resolution (.25, .5, and 1 degree)
CAM almost always gating factor in run times
Data transfer rates minimal
5 data values CAM to SWAT
1 data value SWAT to CAM

21. Future Tasks Additional SWAT configurations for larger scales
Readying the system for scientific experimentation
Currently working on replacement of CAM with WRF
Abstraction of data exchange within the ESMF wrapper code to accommodate configuration of different variables for different model implementations
Setting up SWAT as a service (running on Windows platform) so both models can be remotely accessible
Exploration of cloud computing services