1. ARGONNE NATIONAL LABORATORY Climate Modeling on the Jazz Linux Cluster at ANL John Taylor Mathematics and Computer Science & Environmental Research Divisions Argonne National Laboratory and Computation Institute University of Chicago

2. ARGONNE NATIONAL LABORATORY Outline u A description of the Jazz Linux cluster at Argonne National Laboratory u Porting and performance of climate codes on the Jazz Linux cluster –Community Climate Systems Model, CCSM 2.0.1 –Community Atmosphere Model, CAM 2.0.2 –Mesocale Meteorological Model, MM5v3.4 u Regional climate modeling studies at ANL –Long climate simulations on Jazz using MM5v3.6 –Tools for regional climate modeling

3. ARGONNE NATIONAL LABORATORY ANL Jazz Linux Cluster u Compute - 350 nodes, each with a 2.4 GHz Pentium Xeon u Memory - 175 nodes with 2 GB of RAM, 175 nodes with 1 GB of RAM u Storage - 20 TB of clusterwide disk: 10 TB GFS and 10 TB PVFS u Network - Myrinet 2000 u Linpack benchmark - ~ 1 TFLOP

4. ARGONNE NATIONAL LABORATORY Community Climate Systems Model - CCSM 2.0.1 u Download of standard release of CCSM 2.0.1 from NCAR web site u Current mpich release is compatible with multiple executables multiple data model used by CCSM u Build process needs modification e.g. to use mpif90 and mpicc wrappers u Environment variables must be included in shell u Use pgf90 compiler

5. ARGONNE NATIONAL LABORATORY Community Climate Systems Model - CCSM 2.0.1 u CCSM 2.0.1 runs well on Jazz – now at 3 years per wallclock day u Load balance could be further optimized on Jazz u CCSM 2.1 will include build modifications used to run CCSM on Jazz

6. ARGONNE NATIONAL LABORATORY Community Atmosphere Model CAM 2.0.2 u Download of standard release of CAM 2.0.2 from NCAR web site u Makefile needs modification to use mpif90 and mpicc wrappers u Switch on 2-D finite volume dynamics u Assessed performance using 64,92,128,184 processors

7. ARGONNE NATIONAL LABORATORY Cam 2.0.2 With 2-D Fv Dynamics ARGONNE NATIONAL LABORATORY Acknowledgement: IBM Pwr3 data from Art Mirin, LLNL

8. ARGONNE NATIONAL LABORATORY Performance and Scaling u Performed the standard MM5 benchmark on the Jazz Linux cluster at ANL u Ported MM5 to Intel compilers on IA-32 and IA-64 u Added MPE calls to facilitate profiling on Jazz, TeraGrid, etc

9. ARGONNE NATIONAL LABORATORY MM5 Benchmark on Jazz at ANL Source: John Michalakes, NCAR

10. ARGONNE NATIONAL LABORATORY MM5 over Myrinet on Jazz

11. ARGONNE NATIONAL LABORATORY MM5 over Ethernet on Jazz

12. ARGONNE NATIONAL LABORATORY Regional Climate Modeling u Parallel regional climate model development and testing based on MM5v3.6  WRF u Contributing to PIRCS experiments –PIRCS 1b and currently PIRCS 1c 15 year run u Downscaling using boundary and initial conditions derived from high resolution CCM runs made at LLNL

13. ARGONNE NATIONAL LABORATORY Regional Climate Modeling u Testbed for regional climate simulation laboratory – Espresso interface u Delivering regional climate data using interactive web based tools u Performance testing and porting to the NSF TeraGrid

14. ARGONNE NATIONAL LABORATORY PIRCS 1-B EXPERIMENT  We are using Version 3 of the Penn State / NCAR MM5, with the OSU land surface model  Total precipitation results for the period June 1-July 31, 1993 are shown in the center panel  Note the agreement with both the NCEP reanalysis forcing data (left panel) and the NCDC half-degree Cressman analysis of observations (right panel).  We plan to use this experiment and the PIRCS 1a (1988 US drought) as primary test beds for further enhancements of model physics

15. ARGONNE NATIONAL LABORATORY PIRCS-1c 1987 Precipitation

16. ARGONNE NATIONAL LABORATORY PIRCS-1c June 1988 Temps  We are using Version 3 of the Penn State / NCAR MM5 at 52km grid resolution, with the OSU land surface model NCEP I boundary and initial condition data

17. ARGONNE NATIONAL LABORATORY PIRCS-1c 1988 Precipitation

18. ARGONNE NATIONAL LABORATORY PIRCS-1c 1989 Precipitation

19. ARGONNE NATIONAL LABORATORY Large modeling systems are difficult to configure and run Running complex scientific models can require substantial computing skills Managing the computer science reduces the time available for doing science and limits what is possible e.g. MM5 requires many jobs to be submitted to setup and perform a one year run Current approaches are prone to error (especially where the build process is complex) Espresso Motivation

20. ARGONNE NATIONAL LABORATORY Contemporary software tools are not being exploited e.g. Java, XML, Globus Toolkit distributed computing, etc… Provide secure access to remote supercomputing resources Motivation (Cont.)

21. ARGONNE NATIONAL LABORATORY Develop a flexible graphical user interface (GUI) with low maintenance and development costs Incorporate modern software tools in order to dramatically increase flexibility and efficiency while reducing the chance of operator error = Espresso ! Approach

22. ARGONNE NATIONAL LABORATORY Espresso Modeling Interface

23. ARGONNE NATIONAL LABORATORY CO 2 Sensitivity 690-345 ppm July 1986

24. ARGONNE NATIONAL LABORATORY CO 2 Sensitivity 690-345 ppm December 1986

25. ARGONNE NATIONAL LABORATORY Interactive Data Analysis

26. ARGONNE NATIONAL LABORATORY Interactive Data Analysis

27. ARGONNE NATIONAL LABORATORY Conclusions…. u Key climate modeling codes, CCSM, CAM, MM5v3 are performing well on the Jazz Linux cluster –Multi-year regional climate simulations can be achieved on existing IA-32 Linux supercomputers u Future –NSF TeraGrid (IA-64) –WACCM model with Atmospheric Chemistry code –Performing downscaling using high resolution global GCM data

28. ARGONNE NATIONAL LABORATORY Argonne Climate Modeling Group http://www-climate.mcs.anl.gov