Model Coupling Environmental Library

Goals Develop a framework where geophysical models can be easily coupled together –Work across multiple platforms, operating systems –Simple to use Minimal API Easy to add functionality –Minimally intrusive into existing code –Work for both one and two way coupling –Flexible coupling scheduling –“Reasonably” high performance

Target User Base MCEL was designed to be used at the DoD research laboratories and operational centers –Oceanographers and meteorologists, not computer scientists. Simplicity over functionality! –FORTRAN 77 is a modern language MCEL incorporates a simple FORTRAN and C API –29 Total commands –OOPs languages can access MCEL directly Typically requires about 100 additional lines of code to legacy model to couple using MCEL

Current Approach Server/client approach –Centralized server(s) stores coupling information –Clients (numerical models) store and retrieve data from server Data passes through filter(s) between server and client –Filters alter data into ready to use form for clients –Interpolation, data combination, physics… –Filters not tied to any models, at most tied to numerics/physics – Reusable Server archives coupling data, bunches of netCDF files which can be relocated and restarted –Dedicated thread to file output, run at a lower priority not to slow services –Once data is stored, it CAN be scrubbed from memory, scrubbing handled by dedicated thread

Typical Example Ocean Rad Stress Waves Surface Forcing Waves Server Atmospheric Wave Winds Wind/Wave Energy xfer Winds SST Interpolation SST Atmospheric Interpolation

Communication Details Communication handled through CORBA –Cross platform –Free –Multi language support (C++, Java, Python …) MCEL API has FORTRAN/C bindings, object hiding –Security through SSL –Passive waits available Over-subscription of CPUs –Components can come and go as needed –Built in multithreaded support

Communication Infrastructure

Component Details Currently, all processes initiated by the components Models are required to “register” themselves with the coupling framework –Define grid and variable information Grids can be either one, two or three dimensional: regular, structured or unstructured: Lat/Long or Cartesian Coordinates Variables can be one of the standard MPI types Models then set up filter tree Models periodically store data to server Models periodically request data from filter(s) Models use standard time stamp to mark all data

Component Details Cont. Components are all generic, coupling is performed through variable name matching Components can use any parallelization scheme –MPI, OpenMP, Vector, pthreads… –Each component process/thread can communicate independently or through the master Component N to M communication results in N-1-M communication. However, 1 is multithreaded and work is ongoing to fix this API allows data to be pre-fetched in a non-blocking way Concept: On cluster of SMP machines about looking in a local cache first before going to the server

Data Shipment Upon model registration, grid and variable information is passed around between components, filters and servers During operation, only the raw data and variable/temporal information is passed around –Unless a model has had a grid change, then new grid info is distributed Filter outputs available to be shared if still in local cache Caching of filter inputs being incorporated –MPI application requesting the same data for each task

Filter Details Filters allow the encapsulation of physics or numerics into a reusable package such as interpolation Three basic functions, initialize, evaluate and shutdown Inherited off virtual C++ base class which include all the basic MCEL communication infrastructure –Just supply your filter numerics in any language Filters created in “factories” and can created and destroyed as required Filters are multithreaded Filters can have multiple parents and multiple children –Compute once, share to many –Cache flushed on subsequent calls

Performance Details Results are for a 512x512 grid. If interpolation is used, going to an unstructured grid with 40K points –MCEL storage and retrievals typically on the order of five milliseconds (No filters used) for a single process on the same node as the server –MCEL storage and retrievals typically on the order of ten milliseconds (No filters used) for a single process on a different node as the server (Compaq or IBM) –MCEL storage typically on the order of thirty milliseconds (No filters used) for a 32 process on different nodes as the server (Compaq or IBM) –MCEL retrieval time through an interpolation filter for a 32 process parallel job requires typically 0.1 to 0.2 seconds

Models Currently MCEL Aware ADCIRC ADH COAMPS HYCOM LSOM NCOM REF/DIF STWAVE SWAN WAM WaveWatch WRF

Closing MCEL designed with legacy codes in mind and to be as easy to use as possible MCEL reasonable for coupling every timestep at the most –Excessive overhead for inner-loop coupling MCEL is in alpha release MCEL is publicly available via anonymous CVS MCEL tutorial available at the DoD HPCMO UGC meeting in June