ARGONNE NATIONAL LABORATORY Climate Modeling on the Jazz Linux Cluster at ANL John Taylor Mathematics and Computer Science & Environmental Research Divisions Argonne National Laboratory and Computation Institute University of Chicago

ARGONNE NATIONAL LABORATORY Outline u A description of the Jazz Linux cluster at Argonne National Laboratory u Porting and performance of climate codes on the Jazz Linux cluster –Community Climate Systems Model, CCSM –Community Atmosphere Model, CAM –Mesocale Meteorological Model, MM5v3.4 u Regional climate modeling studies at ANL –Long climate simulations on Jazz using MM5v3.6 –Tools for regional climate modeling

ARGONNE NATIONAL LABORATORY ANL Jazz Linux Cluster u Compute nodes, each with a 2.4 GHz Pentium Xeon u Memory nodes with 2 GB of RAM, 175 nodes with 1 GB of RAM u Storage - 20 TB of clusterwide disk: 10 TB GFS and 10 TB PVFS u Network - Myrinet 2000 u Linpack benchmark - ~ 1 TFLOP

ARGONNE NATIONAL LABORATORY Community Climate Systems Model - CCSM u Download of standard release of CCSM from NCAR web site u Current mpich release is compatible with multiple executables multiple data model used by CCSM u Build process needs modification e.g. to use mpif90 and mpicc wrappers u Environment variables must be included in shell u Use pgf90 compiler

ARGONNE NATIONAL LABORATORY Community Climate Systems Model - CCSM u CCSM runs well on Jazz – now at 3 years per wallclock day u Load balance could be further optimized on Jazz u CCSM 2.1 will include build modifications used to run CCSM on Jazz

ARGONNE NATIONAL LABORATORY Community Atmosphere Model CAM u Download of standard release of CAM from NCAR web site u Makefile needs modification to use mpif90 and mpicc wrappers u Switch on 2-D finite volume dynamics u Assessed performance using 64,92,128,184 processors

ARGONNE NATIONAL LABORATORY Cam With 2-D Fv Dynamics ARGONNE NATIONAL LABORATORY Acknowledgement: IBM Pwr3 data from Art Mirin, LLNL

ARGONNE NATIONAL LABORATORY Performance and Scaling u Performed the standard MM5 benchmark on the Jazz Linux cluster at ANL u Ported MM5 to Intel compilers on IA-32 and IA-64 u Added MPE calls to facilitate profiling on Jazz, TeraGrid, etc

ARGONNE NATIONAL LABORATORY MM5 Benchmark on Jazz at ANL Source: John Michalakes, NCAR

ARGONNE NATIONAL LABORATORY MM5 over Myrinet on Jazz

ARGONNE NATIONAL LABORATORY MM5 over Ethernet on Jazz

ARGONNE NATIONAL LABORATORY Regional Climate Modeling u Parallel regional climate model development and testing based on MM5v3.6  WRF u Contributing to PIRCS experiments –PIRCS 1b and currently PIRCS 1c 15 year run u Downscaling using boundary and initial conditions derived from high resolution CCM runs made at LLNL

ARGONNE NATIONAL LABORATORY Regional Climate Modeling u Testbed for regional climate simulation laboratory – Espresso interface u Delivering regional climate data using interactive web based tools u Performance testing and porting to the NSF TeraGrid

ARGONNE NATIONAL LABORATORY PIRCS 1-B EXPERIMENT  We are using Version 3 of the Penn State / NCAR MM5, with the OSU land surface model  Total precipitation results for the period June 1-July 31, 1993 are shown in the center panel  Note the agreement with both the NCEP reanalysis forcing data (left panel) and the NCDC half-degree Cressman analysis of observations (right panel).  We plan to use this experiment and the PIRCS 1a (1988 US drought) as primary test beds for further enhancements of model physics

ARGONNE NATIONAL LABORATORY PIRCS-1c 1987 Precipitation

ARGONNE NATIONAL LABORATORY PIRCS-1c June 1988 Temps  We are using Version 3 of the Penn State / NCAR MM5 at 52km grid resolution, with the OSU land surface model NCEP I boundary and initial condition data

ARGONNE NATIONAL LABORATORY PIRCS-1c 1988 Precipitation

ARGONNE NATIONAL LABORATORY PIRCS-1c 1989 Precipitation

ARGONNE NATIONAL LABORATORY Large modeling systems are difficult to configure and run Running complex scientific models can require substantial computing skills Managing the computer science reduces the time available for doing science and limits what is possible e.g. MM5 requires many jobs to be submitted to setup and perform a one year run Current approaches are prone to error (especially where the build process is complex) Espresso Motivation

ARGONNE NATIONAL LABORATORY Contemporary software tools are not being exploited e.g. Java, XML, Globus Toolkit distributed computing, etc… Provide secure access to remote supercomputing resources Motivation (Cont.)

ARGONNE NATIONAL LABORATORY Develop a flexible graphical user interface (GUI) with low maintenance and development costs Incorporate modern software tools in order to dramatically increase flexibility and efficiency while reducing the chance of operator error = Espresso ! Approach

ARGONNE NATIONAL LABORATORY Espresso Modeling Interface

ARGONNE NATIONAL LABORATORY CO 2 Sensitivity ppm July 1986

ARGONNE NATIONAL LABORATORY CO 2 Sensitivity ppm December 1986

ARGONNE NATIONAL LABORATORY Interactive Data Analysis

ARGONNE NATIONAL LABORATORY Interactive Data Analysis

ARGONNE NATIONAL LABORATORY Conclusions…. u Key climate modeling codes, CCSM, CAM, MM5v3 are performing well on the Jazz Linux cluster –Multi-year regional climate simulations can be achieved on existing IA-32 Linux supercomputers u Future –NSF TeraGrid (IA-64) –WACCM model with Atmospheric Chemistry code –Performing downscaling using high resolution global GCM data

ARGONNE NATIONAL LABORATORY Argonne Climate Modeling Group