GMAO Seasonal Forecast ESMF Status Climate NCAR/LANL CCSM Data Assimilation GFDL FMS Suite NASA GMAO Analysis Weather GMAO Seasonal Forecast MITgcm Cecelia DeLuca Interagency Working Group Meeting November 30, 2004 NCEP Forecast

Outline What is ESMF? ESMF Open Source Development Development Status Portability and Performance Funding Status Questions/Discussion

What is ESMF? ESMF provides tools for turning model codes into components with standard interfaces and standard drivers ESMF provides data structures and common utilities that components use to organize codes to improve performance portability for common services such as data communications, regridding, time management and message logging

Why Use ESMF? Interoperability – ESMF enables modeling groups to exchange codes, compare codes, extend codes, and use codes in different contexts Reuse – ESMF prevents redundant development of common utilities and offers robust toolkits with many features and good documentation Ease of Use – ESMF standard component interfaces enable modelers to understand new codes quickly, and consistent look and feel across the entire framework helps users learn new functions quickly Performance Portability – the ESMF virtual machine provides excellent portability, good performance, and a consistent interface to diverse architectures

Application Example: GEOS-5 Each box is an ESMF component Every component has a standard interface so that it is swappable New components can easily be added to the system Data in and out of components are packaged as state types Coupling interfaces are standardized but coupling methods are customizable Coupling tools include regridding and redistribution methods

Outline What is ESMF? ESMF Open Source Development Development Status Portability and Performance Funding Status Questions/Discussion

ESMF is a Community Effort The ESMF collaboration has roots in the grass-roots Common Modeling Infrastructure Working Group, and grew out of Earth system modelers’ desire for shared software Users define development priorities Users actively test and evaluate the framework design and implementation ~15% of ESMF source code is from user contributions (IO from WRF, resource file manager from GMAO, regridding from Los Alamos)

Open Source Development Open source license (GPL) Open repositories: web-browsable CVS repositories for source code and contributions Open development priorities and schedule – priorities set based on user meetings, telecons, and mailing list discussions, web-browsable task lists Open testing: 1000+ tests are bundled with the ESMF distribution and can be run by users Open port status: results of nightly tests on many platforms are web-browsable Open metrics: test coverage, lines of code, requirements status are updated regularly and web-browsable

Open Source Constraints ESMF does not allow unmoderated check-ins to its main CVS repository (though there is minimal check-in oversight for the contributions repository) ESMF has a co-located, line managed Core Team whose members are dedicated to framework implementation and support – it does not rely on a volunteer labor ESMF actively sets priorities based on user needs and feedback ESMF requires that contributions follow project conventions and standards for code and documentation ESMF schedules regular releases and meetings The above are necessary for development to proceed at the pace desired by sponsors and users, and to provide the level of quality and customer support necessary for codes in this domain

Outline What is ESMF? ESMF Open Source Development Development Status Portability and Performance Funding Status Questions/Discussion

ESMF Status Overall architecture well-defined and well-accepted Components, States, Fields, low-level communications mature Logically rectangular grids with regular and arbitrary distributions implemented On-line parallel regridding (bilinear, 1st order conservative) completed and optimized Other parallel methods, e.g. halo, redistribution, low-level comms implemented Utilities such as time manager, logging, and configuration manager usable and adding features Virtual machine with uniform interface to shared / distributed memory implemented, hooks for load balancing implemented

ESMF Near-Term Priorities (through June 2004) Concurrent components Wholly irregular grid distributions Grid merges Optimized regridding and low-level communications, including non-blocking communications More grids (including unstructured, tripolar, displaced pole, spectral, cubed sphere) and more regridding methods (bicubic, 2nd order conservative) Fractional times and real times Multi-file and other options for logging Reworked design and implementation of array / grid / field interface and array-level communications Asynchronous IO and read/write grid specifications

ESMF Long-Term Priorities Improve portability, performance, and error handling, and expand and improve documentation, tutorial materials, and training program Develop and assimilate contributions of new functionality into the ESMF software (nested and adaptive grids, data assimilation support including adjoints, MPMD, improved IO) Modify the collaboration environment and project organization so that it is effective with multiple sponsors and a larger number of collaborators Expand the program of collaboration with CCA, PRISM and other national and international infrastructure initiatives; Begin design and implementation of Earth System Modeling Environment (ESME)

Some Metrics … Core Team currently has 2 testers, .5 performance analyst, 5 developers, 1 admin/web support, 1 manager 273 ESMF interfaces implemented, 250 fully or partially tested, 91% fully or partially tested. 142,000 SLOC, 26,000 lines of text 63 open bugs, ~316 closed bugs 785 downloads

ESMF Current Challenges Quality and correctness of source code, especially numerical methods – recent addition of tester contributed by NASA Goddard Process for design and interface review Lead for development of advanced grids Requirements database and requirements tracking – new software packages being explored Clarity and completeness of documentation and training program materials – in short term, possible addition of casual staff for technical editing

NASA Deliverable Schedule and Metrics Public delivery of prototype ESMF v1.0 in May 2003 Completion of first coupling demonstrations using ESMF in March 2004 Delivered ESMF v2.0 in June 2004 Delivery of ESMF v2.1.0 in December 2004 (includes concurrency) All project codes scheduled to achieve partial adoption (use of the ESMF component layer and coupling) by November 2004 All project codes scheduled to achieve full adoption (use of the component layer and coupling plus 3 or more utilities) by June 2005 Reuse Interoperability Ease of Adoption Performance 15 applications use ESMF component coupling services and 3+ utilities 8 new applications comprised of never-before coupled components 2 codes adopt ESMF with < 2% lines of code changed, or within 120 FTE-hours No more than 10% overhead in time to solution, no degradation in scaling

Outline What is ESMF? ESMF Open Source Development Development Status Portability and Performance Funding Status Questions/Discussion

ESMF Platform Support SGI Altix IBM Compaq Linux (Intel, PGI, NAG, Absoft, Lahey) Mac xlf X1

ESMF Component Overhead Measures overhead of ESMF superstructure in NCEP Spectral Statistical Analysis (SSI), ~1% overall Run on NCAR IBM Runs done by JPL staff, confirmed by NCEP developers

ESMF Regridding Performance, Initialization Comparison with the Argonne Model Coupling Toolkit (MCT) bundled with CCSM Run on NCAR IBM Runs done by JPL staff, not yet confirmed by Argonne developers

ESMF Regridding Performance, Run Time Comparison with the Argonne Model Coupling Toolkit (MCT) bundled with CCSM Run on NCAR IBM Runs done by JPL staff, not yet confirmed by Argonne developers

Outline What is ESMF? ESMF Open Source Development Development Status Portability and Performance Funding Status Questions/Discussion

Shared Overhead Cost ROUTE ROLE(S) TOTAL $/YR Interagency support Program coordinator $60K (.25 FTE) Administrator $120K Deployment coordinators $480K Travel, meetings* *Travel, meeting, and other expenses related to project administration and management; computing time.

NASA ROUTE ROLE(S) RESOURCES Modeling recompetition Development plug-ins 6 FTE research team and management (NASA/open) Application plug-ins 6 FTE application developers (NASA) 1 FTE application support (NASA) Programmatic funding Core development 3 FTE core team for development, integration, support, testing (NCAR)* * Current development indicates that for about six external staff trying to adopt or contribute to ESMF it is useful to have 1 FTE dedicated to development/support/integration (over and above general support) on the core team.

NSF ROUTE ROLE(S) RESOURCES NCAR base funds Core development 1 FTE core team manager (NCAR) 1 FTE development, support, integration, testing (NCAR) ½ FTE web development (NCAR) Application development 1 FTE CCSM (NCAR) New competed program: ESMF for univ. science applications Application plug-ins for universities 6-8 FTE application developers, post-docs (open) 2 core team application support (NCAR) Existing competed programs, e.g. CI, SUGR, ITR, SEIII Research opportunities As awarded

NOAA ROUTE ROLE(S) RESOURCES NOAA base funds Application plug-ins 6 FTE application developers (NOAA) Core development 2 FTE development, support, integration, testing (NCAR) Standards program coordinator 1 FTE standards coordinator (NOAA)

DoD ROUTE ROLE(S) RESOURCES BECI proposal Integrate DoD applications with ESMF 1 FTE WRF (NCAR) 5+ FTE application developers (DoD) Core team support 2 FTE development, support, integration, testing (NCAR) 1 FTE computational scientist (DoD)

DoE ROUTE ROLE(S) RESOURCES Existing DoE project staff Integrate with DoE infrastructure projects (ESG, PCMDI) 2 FTE integration Competed programs, e.g. SciDAC Research and development opportunities As awarded

Resulting Allocation FUNCTION RESOURCES Core development/test/integration/web 5 FTE + 1 core team manager + ½ web Core support and outreach 5 FTE + 1 support manager Development plug-ins 6 FTE NASA, 1 FTE DoD Application plug-ins 6 FTE NASA,1 FTE NCAR, 6 FTE NOAA, 5+ FTE DoD, 6-8 NSF/university (20+ FTE in all) Integration with existing infrastructure efforts 2 FTE Standards development 2 FTE coordinators of Joint Specification Team 1 FTE standards coordinator Program coordinator 1 FTE program coordinator Administrative assistance 1 FTE admin asst

Outline What is ESMF? ESMF Open Source Development Development Status Portability and Performance Funding Status Questions/Discussion