1. GMAO Seasonal Forecast
ESMF Status
Climate
NCAR/LANL CCSM
Data Assimilation
GFDL FMS Suite
NASA GMAO Analysis
Weather
GMAO Seasonal Forecast
MITgcm
Cecelia DeLuca
Interagency Working Group Meeting
November 30, 2004
NCEP Forecast

2. Outline What is ESMF? ESMF Open Source Development Development Status
Portability and Performance
Funding Status
Questions/Discussion

3. What is ESMF?
ESMF provides tools for turning model codes into components with standard interfaces and standard drivers
ESMF provides data structures and common utilities that components use
to organize codes
to improve performance portability
for common services such as data communications, regridding, time management and message logging

4. Why Use ESMF?
Interoperability – ESMF enables modeling groups to exchange codes, compare codes, extend codes, and use codes in different contexts
Reuse – ESMF prevents redundant development of common utilities and offers robust toolkits with many features and good documentation
Ease of Use – ESMF standard component interfaces enable modelers to understand new codes quickly, and consistent look and feel across the entire framework helps users learn new functions quickly
Performance Portability – the ESMF virtual machine provides excellent portability, good performance, and a consistent interface to diverse architectures

5. Application Example: GEOS-5
Each box is an ESMF component
Every component has a standard interface so that it is swappable
New components can easily be added to the system
Data in and out of components are packaged as state types
Coupling interfaces are standardized but coupling methods are customizable
Coupling tools include regridding and redistribution methods

6. Outline What is ESMF? ESMF Open Source Development Development Status
Portability and Performance
Funding Status
Questions/Discussion

7. ESMF is a Community Effort
The ESMF collaboration has roots in the grass-roots Common Modeling Infrastructure Working Group, and grew out of Earth system modelers’ desire for shared software
Users define development priorities
Users actively test and evaluate the framework design and implementation
~15% of ESMF source code is from user contributions (IO from WRF, resource file manager from GMAO, regridding from Los Alamos)

8. Open Source Development
Open source license (GPL)
Open repositories: web-browsable CVS repositories for source code and contributions
Open development priorities and schedule – priorities set based on user meetings, telecons, and mailing list discussions, web-browsable task lists
Open testing: tests are bundled with the ESMF distribution and can be run by users
Open port status: results of nightly tests on many platforms are web-browsable
Open metrics: test coverage, lines of code, requirements status are updated regularly and web-browsable

9. Open Source Constraints
ESMF does not allow unmoderated check-ins to its main CVS repository (though there is minimal check-in oversight for the contributions repository)
ESMF has a co-located, line managed Core Team whose members are dedicated to framework implementation and support – it does not rely on a volunteer labor
ESMF actively sets priorities based on user needs and feedback
ESMF requires that contributions follow project conventions and standards for code and documentation
ESMF schedules regular releases and meetings
The above are necessary for development to proceed at the pace desired by sponsors and users, and to provide the level of quality and customer support necessary for codes in this domain

10. Outline What is ESMF? ESMF Open Source Development Development Status
Portability and Performance
Funding Status
Questions/Discussion

11. ESMF Status Overall architecture well-defined and well-accepted
Components, States, Fields, low-level communications mature
Logically rectangular grids with regular and arbitrary distributions implemented
On-line parallel regridding (bilinear, 1st order conservative) completed and optimized
Other parallel methods, e.g. halo, redistribution, low-level comms implemented
Utilities such as time manager, logging, and configuration manager usable and adding features
Virtual machine with uniform interface to shared / distributed memory implemented, hooks for load balancing implemented

12. ESMF Near-Term Priorities (through June 2004)
Concurrent components
Wholly irregular grid distributions
Grid merges
Optimized regridding and low-level communications, including non-blocking communications
More grids (including unstructured, tripolar, displaced pole, spectral, cubed sphere) and more regridding methods (bicubic, 2nd order conservative)
Fractional times and real times
Multi-file and other options for logging
Reworked design and implementation of array / grid / field interface and array-level communications
Asynchronous IO and read/write grid specifications

13. ESMF Long-Term Priorities
Improve portability, performance, and error handling, and expand and improve documentation, tutorial materials, and training program
Develop and assimilate contributions of new functionality into the ESMF software (nested and adaptive grids, data assimilation support including adjoints, MPMD, improved IO)
Modify the collaboration environment and project organization so that it is effective with multiple sponsors and a larger number of collaborators
Expand the program of collaboration with CCA, PRISM and other national and international infrastructure initiatives;
Begin design and implementation of Earth System Modeling Environment (ESME)

14. Some Metrics …
Core Team currently has 2 testers, .5 performance analyst, 5 developers, 1 admin/web support, 1 manager
273 ESMF interfaces implemented, 250 fully or partially tested, 91% fully or partially tested.
142,000 SLOC, 26,000 lines of text
63 open bugs, ~316 closed bugs
785 downloads

15. ESMF Current Challenges
Quality and correctness of source code, especially numerical methods – recent addition of tester contributed by NASA Goddard
Process for design and interface review
Lead for development of advanced grids
Requirements database and requirements tracking – new software packages being explored
Clarity and completeness of documentation and training program materials – in short term, possible addition of casual staff for technical editing

16. NASA Deliverable Schedule and Metrics
Public delivery of prototype ESMF v1.0 in May 2003
Completion of first coupling demonstrations using ESMF in March 2004
Delivered ESMF v2.0 in June 2004
Delivery of ESMF v2.1.0 in December 2004 (includes concurrency)
All project codes scheduled to achieve partial adoption (use of the ESMF component layer and coupling) by November 2004
All project codes scheduled to achieve full adoption (use of the component layer and coupling plus 3 or more utilities) by June 2005
Reuse
Interoperability
Ease of Adoption
Performance
15 applications use ESMF component coupling services and 3+ utilities
8 new applications comprised of never-before coupled components
2 codes adopt ESMF with < 2% lines of code changed, or within 120 FTE-hours
No more than 10% overhead in time to solution, no degradation in scaling

17. Outline What is ESMF? ESMF Open Source Development Development Status
Portability and Performance
Funding Status
Questions/Discussion

18. ESMF Platform Support SGI Altix IBM Compaq
Linux (Intel, PGI, NAG, Absoft, Lahey)
Mac xlf X1

19. ESMF Component Overhead
Measures overhead of ESMF superstructure in NCEP Spectral Statistical Analysis (SSI), ~1% overall
Run on NCAR IBM
Runs done by JPL staff, confirmed by NCEP developers

20. ESMF Regridding Performance, Initialization
Comparison with the Argonne Model Coupling Toolkit (MCT) bundled with CCSM
Run on NCAR IBM
Runs done by JPL staff, not yet confirmed by Argonne developers

21. ESMF Regridding Performance, Run Time
Comparison with the Argonne Model Coupling Toolkit (MCT) bundled with CCSM
Run on NCAR IBM
Runs done by JPL staff, not yet confirmed by Argonne developers

22. Outline What is ESMF? ESMF Open Source Development Development Status
Portability and Performance
Funding Status
Questions/Discussion

23. Shared Overhead Cost
ROUTE
ROLE(S)
TOTAL $/YR
Interagency support
Program coordinator
$60K (.25 FTE)
Administrator
$120K
Deployment coordinators
$480K
Travel, meetings*
*Travel, meeting, and other expenses related to project administration and management; computing time.

24. NASA ROUTE ROLE(S) RESOURCES Modeling recompetition
Development plug-ins
6 FTE research team and management (NASA/open)
Application plug-ins
6 FTE application developers (NASA)
1 FTE application support (NASA)
Programmatic funding
Core development
3 FTE core team for development, integration, support, testing (NCAR)*
* Current development indicates that for about six external staff trying to adopt or contribute to ESMF it is useful to have 1 FTE dedicated to development/support/integration (over and above general support) on the core team.

25. NSF ROUTE ROLE(S) RESOURCES NCAR base funds Core development
1 FTE core team manager (NCAR)
1 FTE development, support, integration, testing (NCAR)
½ FTE web development (NCAR)
Application development
1 FTE CCSM (NCAR)
New competed program: ESMF for univ. science applications
Application plug-ins for universities
6-8 FTE application developers, post-docs (open)
2 core team application support (NCAR)
Existing competed programs, e.g. CI, SUGR, ITR, SEIII
Research opportunities
As awarded

26. NOAA ROUTE ROLE(S) RESOURCES NOAA base funds Application plug-ins
6 FTE application developers (NOAA)
Core development
2 FTE development, support, integration, testing (NCAR)
Standards program coordinator
1 FTE standards coordinator (NOAA)

27. DoD ROUTE ROLE(S) RESOURCES BECI proposal
Integrate DoD applications with ESMF
1 FTE WRF (NCAR)
5+ FTE application developers (DoD)
Core team support
2 FTE development, support, integration, testing (NCAR)
1 FTE computational scientist (DoD)

28. DoE ROUTE ROLE(S) RESOURCES Existing DoE project staff
Integrate with DoE infrastructure projects (ESG, PCMDI)
2 FTE integration
Competed programs, e.g. SciDAC
Research and development opportunities
As awarded

29. Resulting Allocation FUNCTION RESOURCES
Core development/test/integration/web
5 FTE + 1 core team manager + ½ web
Core support and outreach
5 FTE + 1 support manager
Development plug-ins
6 FTE NASA, 1 FTE DoD
Application plug-ins
6 FTE NASA,1 FTE NCAR, 6 FTE NOAA, 5+ FTE DoD, 6-8 NSF/university (20+ FTE in all)
Integration with existing infrastructure efforts
2 FTE
Standards development
2 FTE coordinators of Joint Specification Team
1 FTE standards coordinator
Program coordinator
1 FTE program coordinator
Administrative assistance
1 FTE admin asst

30. Outline What is ESMF? ESMF Open Source Development Development Status
Portability and Performance
Funding Status
Questions/Discussion