1. Höchstleistungsrechenzentrum Stuttgart Matthias M üller SPEC Benchmarks for Large Systems Matthias Mueller, Kumaran Kalyanasundaram, G. Gaertner, W. Jones, R. Eigenmann, R. Lieberman, M. van Waveren, and B. Whitney SPEC High Performance Group mueller@hlrs.de

2. Höchstleistungsrechenzentrum Stuttgart Matthias M üller What is SPEC? The Standard Performance Evaluation Corporation (SPEC) is a non-profit corporation formed to establish, maintain and endorse a standardized set of relevant benchmarks that can be applied to the newest generation of high-performance computers. SPEC develops suites of benchmarks and also reviews and publishes submitted results from our member organizations and other benchmark licensees.

3. Höchstleistungsrechenzentrum Stuttgart Matthias M üller SPEC High-Performance Group Founded in 1994 Mission: To establish, maintain, and endorse a suite of benchmarks that are representative of real-world high-performance computing applications. SPEC/HPG includes members from both industry and academia. Benchmark products: –SPEC OMP (OMPM2001, OMPL2001) –SPEC HPC2002 released at SC 2002

4. Höchstleistungsrechenzentrum Stuttgart Matthias M üller Currently active SPEC HPG Members Fujitsu HP IBM Intel SGI SUN UNISYS University of Purdue University of Stuttgart

5. Höchstleistungsrechenzentrum Stuttgart Matthias M üller Where is SPEC Relative to Other Benchmarks ? There are many metrics, each one has its purpose Raw machine performance: Tflops Microbenchmarks: Stream Algorithmic benchmarks: Linpack Compact Apps/Kernels: NAS benchmarks Application Suites: SPEC User-specific applications: Custom benchmarks

6. Höchstleistungsrechenzentrum Stuttgart Matthias M üller Comparison of different benchmark classes coveragerelevanceIdentify problems Time evolution portability Micro00++++ Algorithmic-0++++ Kernels00+++ SPEC+++++ Apps-++000

7. Höchstleistungsrechenzentrum Stuttgart Matthias M üller SPEC OMP Benchmark suite developed by SPEC HPG (High Performance Group) Benchmark suite for performance testing of shared memory processor systems Uses OpenMP versions of SPEC CPU2000 benchmarks and candidates

8. Höchstleistungsrechenzentrum Stuttgart Matthias M üller SPEC OMP/CPU2000 differences Larger working set sizes, 1.6GB for OMPM2001, 6.5 GB for OMPL2001; it is 200MB for CPU2000 Longer run times (>1000 s/cpu for CPU2000 vs >10,000 s/cpu for OMP2000 medium) Focus on SMP systems, and issued by HPG SPEC OMP based on work for CPU2000, SPEC OMP mixes integer and FP in one suite OMPM is focused on 4-way to 16-way systems OMPL is targeting 32-way and larger systems

9. Höchstleistungsrechenzentrum Stuttgart Matthias M üller SPEC OMP Results 66 submitted results for OMPM 24 submitted results for OMPL VendorHP SUNSGI ArchitectureSuperdome Fire 15KO3800 CPUPA-8700+Itanium2UltraSPARC III R12000 Speed87515001200400 L1 Inst0.75 MB16 KB32 KB L1 Data1.5 MB16 KB64 KB32 KB L2-256 KB8 MB L3-6144 KB--

10. Höchstleistungsrechenzentrum Stuttgart Matthias M üller SPEC OMPL Results: Benchmark with scaling to 128

11. Höchstleistungsrechenzentrum Stuttgart Matthias M üller SPEC OMPL Results: Superlinear scaling of applu

12. Höchstleistungsrechenzentrum Stuttgart Matthias M üller SPEC OMPL Results: Benchmark with scaling to 64

13. Höchstleistungsrechenzentrum Stuttgart Matthias M üller SPEC HPC2002 Benchmark Full Application benchmarks (including I/O) targeted at HPC platforms Serial and parallel (OpenMP and/or MPI) Currently three applications: –SPECenv: weather forecast –SPECseis: seismic processing, used in the search for oil and gas –SPECchem: comp. chemistry, used in chemical and pharmaceutical industries (gamess) All codes include several data sizes

14. Höchstleistungsrechenzentrum Stuttgart Matthias M üller SPECenv execution models on a Sun Fire 6800 Medium scales better OpenMP best for small size MPI best for medium size

15. Höchstleistungsrechenzentrum Stuttgart Matthias M üller SPEC HPC2002 Results: SPECenv scaling

16. Höchstleistungsrechenzentrum Stuttgart Matthias M üller SPECseis execution models on a Sun Fire 6800 Medium scales better OpenMP scales better than MPI

17. Höchstleistungsrechenzentrum Stuttgart Matthias M üller SPEC HPC2002 Results: SPECseis scaling

18. Höchstleistungsrechenzentrum Stuttgart Matthias M üller SPECchem execution models on a Sun Fire 6800 Medium shows better scalability MPI is better than OpenMP

19. Höchstleistungsrechenzentrum Stuttgart Matthias M üller SPEC HPC2002 Results: SPECchem scaling

20. Höchstleistungsrechenzentrum Stuttgart Matthias M üller Hybrid Execution for SPECchem

21. Höchstleistungsrechenzentrum Stuttgart Matthias M üller Current and Future Work SPEC HPC: –Update of SPECchem –Improving portability, including tools –Larger datasets New release of SPEC OMP: –Inclusion of alternative sources –Merge OMPM and OMPL on one CD Adoption of new benchmark codes: –Should represent a type of computation that is regularly performed on HPC systems –We currently examine CPU2004 for candidates

22. Höchstleistungsrechenzentrum Stuttgart Matthias M üller Conclusion and Summary Results of OMPL and HPC2002: –Scalability of many programs to 128 CPUs Best choice of programming model (MPI,OpenMP, hybrid) depends on: –Hardware –Program –Data set size SPEC HPC will continue to update and improve the benchmark suites.