1. CCSM Portability and Performance, Software Engineering Challenges, and Future Targets Tony Craig National Center for Atmospheric Research Boulder, Colorado, USA CAS Meeting, September 7-11, 2003, Annecy France

2. Topics CCSM SE and design overview Coupler design and performance Production and performance –portability –scaling SE Challenges The Future

3. CCSM Overview CCSM = Community Climate System Model (NCAR) Designed to evaluate and understand earth’s global climate, both historical and future. Multiple executables (5) –Atmosphere (CAM), MPI/OpenMP –Ocean (POP), MPI –Land (CLM), MPI/OpenMP –Sea Ice (CSIM), MPI –Coupler (CPL6), MPI

4. CCSM SE Overview Good science top priority Fortran 90 (mostly) 500k lines of code Community project, dozens of developers Collaborations are critical –University Community –DOE - SciDAC –NASA - ESMF Regular code releases Netcdf history files Binary restart files Many levels of parallelism-multiple executables, MPI, OpenMP

5. CCSM “Hub and Spoke” System cpl atm ocnice lnd Each component is a separate executable Each component on a unique set of hardware processors All communications go through coupler Coupler –communicates with all components –maps (interpolates) data –merges fields –computes some fluxes –has diagnostic, history, and restart capability

6. The CCSM coupler Recent redesign (cpl6) Create a fully parallel distributed memory coupler Implement M to N communication between components Improve communication performance to minimize bottlenecks at higher resolutions in the future Improve coupling interfaces, abstract communication method away from components Improve usability, flexibility, and extensibility of coupled system Improve overall performance

7. The Solution MCT* MPH** *Model Coupling Toolkit (DOE Argonne National Lab) ** Multi-Component Handshaking Library (DOE Lawrence Berkley National Lab) cpl6 Build a new coupler framework with abstracted, parallel communication software in the foundation. Create a coupler application instantiation called cpl6 which reproduces the functionality of cpl5:

8. cpl6 Design: Another view of CCSM In cpl5, MPI was the coupling interface In cpl6, the “coupler” is now attached to each component –Components unaware of coupling method –Coupling work can be carried out on component processors –Separate coupler no longer absolutely required atm lnd iceocn cpl coupling interface layer hardware processors

9. CCSM Communication: cpl5 vs cpl6 cpl5 gatherscatter comm (root to root) comm (root to root) copy Coupler on 8pes Ice component on 16pes 240 transfers, 21 fields Production configuration cpl6 comm (M to N) comm (M to N) copy NO copy cpl5 communication=61.5s cpl6 communication=18.5s

10. CCSM Production Forward integration of relatively coarse models –atm/land - T42 (128x64, L26) –ocn/ice - 1 degree (320x384, L40) Finite difference and spectral, explicit and implicit methods, vertical physics, global sums, nearest neighbor communication I/O not a bottleneck (5 Gb / simulated year) Restart capability (750 Mb) Separate harvesting to local mass storage system Auto resubmit

11. CCSM Throughput vs Resolution Atmosphere Resolution Ocean Resolution ProcessorsThroughput* (yrs/day) T31 (3.7 deg)3 deg 6812.0 T42 (2.8 deg)1 deg15210.9 T421 deg120 8.6 T421 deg104 7.2 T85 (1.4 deg)1 deg200 4.0 T170 (0.7 deg)1 deg400 2.0 (estimate) *IBM power4 system, bluesky, as of 9/1/2003

12. CCSM Throughput vs Platform PlatformProcessorsThroughput*(yrs/day) IBM power3 wh (NCAR)104 3.5 IBM power3 nh (NERSC)112 4.0 SGI O3K (LANL)112 4.0 Linux cluster* (ANL)104 4.0-7.0 (estimate) HP/CPQ Alpha (PSC)104 6.0 (estimate) IBM power4 (NCAR)104 7.2 IBM power4 (NCAR)15210.9 NEC Earth Simulator 8020.0 (estimate) Cray X1 (ORNL) 8020.0 (estimate) T42/1 degree atm/ocn resolution * ANL jazz machine, 2.4Ghz Pentium

13. Ocean Model Performance and Scaling **Courtesy of PW Jones, PH Worley, Y Yoshida, JB White III, J Levesque

14. CCSM Component Scaling CCSM2_2_beta08 T42_gx1v3 IBM Power4, bluesky

15. CCSM Load Balance Example 64 ocn 48 atm 16 ice 8 lnd 16 cpl 152 total Seconds per simulated day processors 3.31.2 11.8 4.014.3 2.11.2 21.7 3.01.1.8 21.3 CCSM2_2_beta08 IBM Power4, bluesky

16. Challenges in the Environment (1) Machines often not well balanced –chip speed –interconnect –memory access –cache –I/O –vector capabilities Each machine is “balanced” differently Optimum coding strategy often depends largely on platform Need to develop “flexible” software strategies

17. RISC vs Vector Data layout; index order, data structure layout Floating operation count (if) versus pipelining (masking) Loop ordering and loop structure Vectorization impacts parallelization Memory access, cache blocking, array layouts, array usage Bottom Line (In My Opinion): –Truly effective cache reuse is very hard to achieve on real codes –Sustained performance on some RISC machines is disappointing –Poor vectorization costs an order of magnitude in performance on vector machines –We are now (re-)vectorizing and expect to pay little or no performance penalty on RISC machines

18. Challenges in the Environment (2) Startup and control of multiple executables Compilers and libraries Tools –Debuggers inadequate; multiple executables and MPI/OpenMP parallel models –Timing and profiling tools generally inadequate IBM HPM getting better Jumpshot works well Cray performance tools look promising –Have avoided instrumenting code (risk, robust, #if) –Use print statements and calls to system clock

19. Summary Science top priority, large community project, regular model releases SE improvements continuous, cpl6 is a success Machines change rapidly and are highly variable in architecture Component scaling and CCSM load balance are acceptable (Re)-Vectorization Tools and machine software can present significant challenges

20. Future Increased coupling flexibility –Single executable –Mixed concurrent/serial design Continue to work on scalar and parallel performance in all models Take advantage of libraries/collaborations for performance portability software, more layering, leverage external efforts –NASA/ESMF –DOE/SCIDAC –University Community –Others IBM is still an important production platform for CCSM CCSM is moving onto vector platforms and linux clusters, production capability on these platforms still to be determined

21. THE END