Kernel and Application Code Performance for a Spectral Atmospheric Global Circulation Model on the Cray T3E and IBM SP Patrick H. Worley Computer Science and Mathematics Division Oak Ridge National Laboratory NERSC Users’ Group Meeting Oak Ridge, TN June 6, 2000

… random collection of benchmarks, looking at communication, serial, and parallel performance on the IBM SP and other MPPs at NERSC and ORNL. Alternative Title

v Research sponsored by the Atmospheric and Climate Research Division and the Office of Mathematical, Information, and Computational Sciences, Office of Science, U.S. Department of Energy under Contract No. DE-AC05-00OR22725 with UT-Battelle, LLC. v These slides have been authored by a contractor of the U.S. Government under contract No. DE-AC05-00OR Accordingly, the U.S. Government retains a nonexclusive, royalty-free license to publish or reproduce the published form of this contribution, or allow others to do so, for U.S. Government purposes v Oak Ridge National Laboratory is managed by UT-Battelle, LLC for the United States Department of Energy under Contract No. DE-AC05- 00OR Acknowledgements

Platforms at NERSC v IBM SP 2-way Winterhawk I SMP “wide” nodes with 1 GB memory 200 MHz Power 3 processors with 4 MB L2 cache 1.6 GB/sec node memory bandwidth (single bus) Omega multistage interconnect v SGI/Cray Research T3E-900 Single processor nodes with 256 MB memory 450 MHz Alpha (EV5) with 96 KB L2 cache 1.2 GB/sec node memory bandwidth 3D torus interconnect

Platforms at ORNL v IBM SP 4-way Winterhawk II SMP “thin” nodes with 2 GB memory 375 MHz Power 3-II processors with 8 MB L2 cache 1.6 GB/sec node memory bandwidth (single bus) Omega multistage interconnect v Compaq AlphaServer SC 4-way ES40 SMP nodes with 2 GB memory 667 MHz Alpha 21264a (EV67) processors with 8 MB L2 cache 5.2 GB/sec node memory bandwidth (dual bus) Quadrics “fat tree” interconnect

Other Platforms v SGI / Cray Research Origin 2000 at LANL 128-way SMP node with 32 GB memory 250 MHz MIPS R10000 processors with 4 MB L2 cache NUMA memory subsystem v IBM SP 16-way Nighthawk II SMP node 375 MHz Power3-II processors with 8 MB L2 cache switch-based memory subsystem Results obtained using prerelease hardware and software

Topics v Interprocessor communication performance v Serial performance PSTSWM spectral dynamics kernel CRM column physics kernel v Parallel performance CCM/MP-2D atmospheric global circulation model

Communication Tests v Interprocessor communication performance within an SMP node between SMP nodes with and without contention with and without cache invalidation for both bidirectional and unidirectional communication protocols v Brief description of some results. For more details, see

Communication Tests MPI_SENDRECV bidirectional and MPI_SEND/MPI_RECV unidirectional bandwidth between nodes on the IBM SP at NERSC

Communication Tests MPI_SENDRECV bidirectional and MPI_SEND/MPI_RECV unidirectional bandwidth between nodes on the IBM SP at NERSC

Communication Tests MPI_SENDRECV bidirectional and MPI_SEND/MPI_RECV unidirectional bandwidth between nodes on the IBM SP at ORNL

Communication Tests Bidirectional bandwidth comparison across platforms: swap between processors 0-1

Communication Tests Bidirectional bandwidth comparison across platforms: swap between processors 0-4

Communication Tests Bidirectional bandwidth comparison across platforms: simultaneous swap between processors 0-4,1-5,2-6,3-7

Communication Tests Bidirectional bandwidth comparison across platforms: 8 processor send/recv ring

Communication Tests v Summary Decent intranode performance is possible. Message passing functionality is good. Switch / NIC performance is limiting in internode communication. Contention for Switch/NIC bandwidth in SMP nodes can be significant.

Serial Performance v Issues Compiler optimization Domain decomposition Memory contention in SMP nodes v Kernel codes PSTSWM - spectral dynamics CRM - column physics

Spectral Dynamics v PSTSWM solves the nonlinear shallow water equations on a sphere using the spectral transform method 99% of floating point operations are fmul, fadd, or fmadd accessing memory linearly, but not much reuse (longitude, vertical, latitude) array index ordering u computation independent between horizontal layers (fixed vertical index) u as vertical dimension size increases, demands on memory increase

Spectral Dynamics PSTSWM on the IBM SP at NERSC Horizontal Resolutions T5: 8x16 T10:16x32 T21:32x64 T42:64x128 T85:128x256 T170:256x512

Spectral Dynamics PSTSWM on the IBM SP at NERSC

Spectral Dynamics PSTSWM Platform comparisons - 1 processor per SMP node Horizontal Resolutions T5: 8x16 T10:16x32 T21:32x64 T42:64x128 T85:128x256 T170:256x512

Spectral Dynamics PSTSWM Platform comparisons - all processors active in SMP node (except Origin-250) Horizontal Resolutions T5: 8x16 T10:16x32 T21:32x64 T42:64x128 T85:128x256 T170:256x512

Spectral Dynamics PSTSWM Platform comparisons - 1 processor per SMP node

Spectral Dynamics PSTSWM Platform comparisons - all processors active in SMP node (except Origin-250)

Spectral Dynamics v Summary Math libraries and relaxed mathematical semantics improve performance significantly on the IBM SP. Node memory bandwidth is important (for this kernel code), especially on bus-based SMP nodes. The IBM SP serial performance is a significant improvement over the (previous generation) Origin and T3E systems.

Column Physics v CRM Column Radiation Model extracted from the Community Climate Model 6% of floating point operations are sqrt, 3% are fdiv exp, log, and pow are among top six most frequently called functions (longitude, vertical, latitude) array index ordering u computations independent between vertical columns (fixed longitude, latitude) u as longitude dimension size increases, demands on memory increase

Column Physics CRM on the NERSC SP longitude-vertical slice, with varying number of longitudes

Column Physics CRM longitude-vertical slice, with varying number of longitudes 1 processor per SMP node

Column Physics v Summary Performance is less sensitive to node memory bandwidth for this kernel code. Performance on the IBM SP is very sensitive to compiler optimization and domain decomposition.

Parallel Performance v Issues Scalability Overhead growth and analysis v Codes CCM/MP-2D

v Message-passing parallel implementation of the National Center for Atmospheric Research (NCAR) Community Climate Model v Computational Domains Physical Domain: Longitude x Latitude x Vertical levels Fourier Domain: Wavenumber x Latitude x Vertical levels Spectral Domain: (Wavenumber x Polynomial degree) x Vertical levels.

CCM/MP-2D v Problem Sizes T42L x 64 x 18 physical domain grid 42 x 64 x 18 Fourier domain grid 946 x 18 spectral domain grid ~59.5 GFlops per simulated day T170L x 256 x 18 physical domain grid 170 x 256 x 18 Fourier domain grid x 18 spectral domain grid ~3231 GFlops per simulated day

CCM/MP-2D v Computations Column Physics u independent between vertical columns Spectral Dynamics u Fourier transform in longitude direction u Legendre transform in latitude direction u tendencies for timestepping calculated in spectral domain, independent between spectral coordinates Semi-Lagrangian Advection u Use local approximations to interpolate wind fields and particle distributions away from grid points.

CCM/MP-2D v Decomposition across latitude u parallelizes the Legendre transform: Use distributed global sum algorithm currently u requires north/south halo updates for semi-Lagrangian advection v Decomposition across longitude u parallelizes the Fourier transform: Either use distributed FFT algorithm or transpose fields and use serial FFT u requires east/west halo updates for semi-Lagrangian advection u requires night/day vertical column swaps to load balance physics

CCM/MP-2D Sensitivity of message volume to domain decomposition

Scalability CCM/MP-2D T42L18 Benchmark

Scalability CCM/MP-2D T170L18 Benchmark

Overhead CCM/MP-2D T42L18 Benchmark Overhead Time Diagnosis

Overhead CCM/MP-2D T170L18 Benchmark Overhead Time Diagnosis

CCM/MP-2D v Summary Parallel algorithm optimization is (still) important for achieving peak performance Bottlenecks u Message-passing bandwidth and latency u SMP node memory bandwidth on the SP