1. page 1 Supercomputing, quo Vadis? Erfahrungen aus 15 Jahren TOP500 und 30 Listen Hans Werner Meuer hans@meuer.de Universität Mannheim & Prometeus GmbH Jahresabschluss-Kolloquium Jülich Supercomputing Centre (JSC) Mittwoch, 19. Dezember 2007

2. JSC Jahresabschluss-Kolloquium 2007 page 2 Herzlichen Glückwunsch

3. JSC Jahresabschluss-Kolloquium 2007 page 3 30th List: The TOP10 ManufacturerComputer Rmax [TF/s] Installation SiteCountryYear#Cores 1IBM BlueGene/L eServer Blue Gene 478.2DOE/NNSA/LLNLUSA2007212,992 2IBM JUGENE BlueGene/P Solution 167.3 Forschungszentrum Juelich Germany200765,536 3SGI SGI Altix ICE 8200 126.9 New Mexico Computing Applications Center USA200714,336 4HP Cluster Platform 3000 BL460c 117.9 Computational Research Laboratories, TATA SONS India200714,240 5HP Cluster Platform 3000 BL460c 102.8 Swedish Government Agency Sweden200713,728 6Sandia/Cray Red Storm Cray XT3 102.2DOE/NNSA/SandiaUSA200626,569 7Cray Jaguar Cray XT3/XT4 101.7DOE/ORNLUSA200723,016 8IBM BGW eServer Blue Gene 91.29IBM Thomas WatsonUSA200540,960 9Cray Franklin Cray XT4 85.37NERSC/LBNLUSA200719,320 10IBM New York Blue eServer Blue Gene 82.16Stony Brook/BNLUSA200736,864

4. JSC Jahresabschluss-Kolloquium 2007 page 4 Mannheim Supercomputer Statistics & Top500 Project Start in 1993 Competition between Manufacturers, Countries and Sites My Supercomputer Favorite in the Top500 Lists The 30 th List as of November 2007 Performance Development and Projection Bell‘s Law Supercomputing, quo vadis? - in Jülich, Germany and Europe - in the Rest of the World Top500, quo vadis? Outline

5. JSC Jahresabschluss-Kolloquium 2007 page 5 Mannheim Supercomputer Statistics & Top500 Project Start in 1993 Mannheim Supercomputer Statistics 1986 – 1992 Presented at ISC’86 – ISC’92/Mannheim Supercomputer Seminars Counting of Supercomputers in the World 530 systems in the year 1992

6. JSC Jahresabschluss-Kolloquium 2007 page 6 Manufacturer Share 0% 20% 40% 60% 80% 100% 1986198719881989199019911992 NEC Hitachi Fujitsu CDC Cray Mannheim Supercomputer Statistics & Top500 Project Start in 1993

7. JSC Jahresabschluss-Kolloquium 2007 page 7 Countries Share 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1986198719881989199019911992 Others Others Europe Germany France UK Japan USA Mannheim Supercomputer Statistics & Top500 Project Start in 1993

8. JSC Jahresabschluss-Kolloquium 2007 page 8 Mannheim Supercomputer Statistics & Top500 Project Start in 1993 Deficits of the Mannheim Supercomputer Statistics Not very reliable Vector Supercomputers only – „numerical“/ mainly „scientific and engineering“ applications Definition of „Supercomputer“ necessary

9. JSC Jahresabschluss-Kolloquium 2007 page 9 Mannheim Supercomputer Statistics & Top500 Project Start in 1993 Top500 Procedure Listing the 500 most powerful computers in the world Yardstick: Rmax of Linpack Solve Ax=b, dense problem, matrix is random Update twice a year: ISC’xy in June in Germany SC’xy in November in the U.S. All information available from the TOP500 webserver at: www.top500.org

10. JSC Jahresabschluss-Kolloquium 2007 page 10 Mannheim Supercomputer Statistics & Top500 Project Start in 1993 Top500 Authors Project was started in spring 1993 by: Hans W. MeuerErich Strohmaier Jack Dongarra Authors since: O6/1993 Horst Simon 11/2000

11. JSC Jahresabschluss-Kolloquium 2007 page 11 „CRPC researcher Jack Dongarra of the University of Tennessee and Oak Ridge National Laboratory is one of three renowned computer scientists who assemble the legendary TOP500 List of the world's fastest supercomputers. Released twice a year since 1993, the list features sites with the most powerful computer systems, determined with information from a questionnaire sent to high-performance computer (HPC) experts, computational scientists, manufacturers, and the Internet community at large. Dongarra, Hans Meuer of the University of Mannheim, and Erich Strohmaier of the University of Tennessee released their June 1999 TOP500 List at the 14th Mannheim Supercomputing Conference and Seminar, held June 10-12 in Mannheim, Germany.“ Source: http://www.crpc.rice.edu/WhatsNew/top500.html The Center for Research on Parallel Computation at Rice University Dongarra's TOP500 List of World's Fastest Supercomputers Released at Mannheim Conference Mannheim Supercomputer Statistics & Top500 Project Start in 1993

12. JSC Jahresabschluss-Kolloquium 2007 page 12 Mannheim Supercomputer Statistics & Top500 Project Start in 1993  Competition between Manufacturers, Countries and Sites My Supercomputer Favorite in the Top500 Lists The 30 th List as of November 2007 Performance Development and Projection Bell‘s Law Supercomputing, quo vadis? - in Jülich, Germany and Europe - in the Rest of the World Top500, quo vadis? Outline

13. JSC Jahresabschluss-Kolloquium 2007 page 13 Top500 Status 1 st Top500 List in June 1993 at ISC’93 in Mannheim 29 th Top500 List on June 27, 2007 at ISC’07 in Dresden 30 th Top500 List on November 13, 2007 at SC07 in Reno 31 st Top500 List on June 18, 2008 at ISC’08 in Dresden 32 nd Top500 List on November 18, 2008 at SC08 in Austin 33 rd Top500 List on June 24, 2009 at ISC’09 in Hamburg Acknowledged by HPC-users, manufacturers and media Competition between Manufacturers, Countries and Sites

14. JSC Jahresabschluss-Kolloquium 2007 page 14 CountriesCountShare % USA22545.00 % Japan11122.20 % Germany5911.80 % France265.20 % United Kingdom255.00 % Australia91.80 % Italy61.20 % Netherlands61.20 % Switzerland40.80 % Canada30.60 % Denmark30.60 % Korea30.60 % Others204.00 % Totals500100 % 1 st List as of 06/1993 100 %500Totals 11.00 %55Others 1.80 %9India 2.00 %10China 0.20 %1Korea 0.20 %1Denmark 1.00 %5Canada 1.40 %7Switzerland 1.20 %6Netherlands 1.20 %6Italy 0.20 %1Australia 9.60 %48United Kingdom 3.40 %17France 6.20 %31Germany 4.00 %20Japan 56.60%283USA Share %CountCountries 30 th List as of 11/2007 Competition between Manufacturers, Countries and Sites

15. JSC Jahresabschluss-Kolloquium 2007 page 15 Competition between Manufacturers, Countries and Sites Countries/Systems

16. JSC Jahresabschluss-Kolloquium 2007 page 16 ManufacturersCountShare % Cray Research20541.00 % Fujitsu6913.80 % Thinking Machines5410.80 % Intel448.80 % Convex367.20 % NEC326.40 % Kendall Square Research 214.20 % MasPar183.60 % Meiko91.80 % Hitachi61.20 % Parsytec30.60 % nCube30.60 % Totals500100 % 1 st List as of 06/1993 7.00 %35Others 100 %500Totals 4.80 %24Dell 4.40 %22SGI 46.40 %232IBM ——nCube ——Parsytec 0.20 %1Hitachi/Fujitsu ——Meiko ——MasPar —— Kendall Square Research 0.40 %2NEC 33.20 %166Hewlett-Packard 0.20 %1Intel ——Thinking Machines 0.60 %3Fujitsu 2.80 %14Cray Inc. Share %CountManufacturers 30 th List as of 11/2007 Competition between Manufacturers, Countries and Sites

17. JSC Jahresabschluss-Kolloquium 2007 page 17 Competition between Manufacturers, Countries and Sites Manufacturer/Systems

18. JSC Jahresabschluss-Kolloquium 2007 page 18 Top Sites through 30 Lists Competition between Manufacturers, Countries and Sites RankSiteCountry% over time 1Lawrence Livermore National LaboratoryUnited States5.39 2Sandia National LaboratoriesUnited States3.70 3Los Alamos National LaboratoryUnited States3.41 4GovernmentUnited States3.34 5The Earth Simulator CenterJapan1.99 6National Aerospace Laboratory of JapanJapan1.70 7Oak Ridge National LaboratoryUnited States1.39 8NCSAUnited States1.31 9NASA/Ames Research Center/NASUnited States1.25 10University of TokyoJapan1.21 11NERSC/LBNLUnited States1.19 12Pittsburgh Supercomputing CenterUnited States1.15 13Semiconductor Company (C)United States1.11 14Naval Oceanographic Office (NAVOCEANO)United States1.08 15ECMWFUnited Kingdom1.02 16ERDC MSRCUnited States0.91 17IBM Thomas J. Watson Research CenterUnited States0.86 18Forschungszentrum Juelich (FZJ)Germany0.84 19Japan Atomic Energy Research InstituteJapan0.83 20Minnesota Supercomputer CenterUnited States0.74

19. JSC Jahresabschluss-Kolloquium 2007 page 19 Mannheim Supercomputer Statistics & Top500 Project Start in 1993 Competition between Manufacturers, Countries and Sites  My Supercomputer Favorite in the Top500 Lists The 30 th List as of November 2007 Performance Development and Projection Bell‘s Law Supercomputing, quo vadis? - in Jülich, Germany and Europe - in the Rest of the World Top500, quo vadis? Outline

20. JSC Jahresabschluss-Kolloquium 2007 page 20 ManufacturerComputerRmax [GF/s] Installation SiteCountryYear#Proc 1IntelASCI Red1068Sandia National Laboratories USA19967264 2HitachiCP-PACS/2048368.2Center for Computational Science Japan19962048 3FujitsuNumerical Wind Tunnel 229National Aerospace Laboratory Japan1996167 4HitachiSR2201/1024220.4University of TokyoJapan19961024 5CrayT3E176Forschungszentrum Jülich Germany1996512 6CrayT3E176GovernmentUSA1997512 7CrayT3E176Max-Planck-Gesell. MPI/IPP Germany1997512 8CrayT3E176NASA/Goddard Space Flight Center USA1996512 9CrayT3E176Pittsburgh Supercomp. Center USA1996512 10CrayT3E176University StuttgartGermany1996512 My Supercomputer Favorite in the Top500 Lists Top10 in the 06/1997 list

21. JSC Jahresabschluss-Kolloquium 2007 page 21 My Supercomputer Favorite in the Top500 Lists Performance [GFlop/s] N=1 N=500 Sum 1 TFlop/s 1 PFlop/s Intel ASCI Red Jun 05

22. JSC Jahresabschluss-Kolloquium 2007 page 22 Number of Teraflop/s Systems in the Top500 1 10 100 500 Jun-97Jun-98Jun-99Jun-00Jun-01Jun-02Jun-03Jun-04Jun-05 My Supercomputer Favorite in the Top500 Lists

23. JSC Jahresabschluss-Kolloquium 2007 page 23 My Supercomputer Favorite in the Top500 Lists

24. JSC Jahresabschluss-Kolloquium 2007 page 24 Mannheim Supercomputer Statistics & Top500 Project Start in 1993 Competition between Manufacturers, Countries and Sites My Supercomputer Favorite in the Top500 Lists  The 30 th List as of November 2007 Performance Development and Projection Bell‘s Law Supercomputing, quo vadis? - in Jülich, Germany and Europe - in the Rest of the World Top500, quo vadis? Outline

25. JSC Jahresabschluss-Kolloquium 2007 page 25 The 30 th List as of November 2007 Processor Architecture/Systems

26. JSC Jahresabschluss-Kolloquium 2007 page 26 The 30 th List as of November 2007 Operating Systems/Systems

27. JSC Jahresabschluss-Kolloquium 2007 page 27 The 30 th List as of November 2007 Processor Generation/Systems (November 2007)

28. JSC Jahresabschluss-Kolloquium 2007 page 28 The 30 th List as of November 2007 Interconnect Family/Systems

29. JSC Jahresabschluss-Kolloquium 2007 page 29 The 30 th List as of November 2007 Architectures / Systems

30. JSC Jahresabschluss-Kolloquium 2007 page 30 Mannheim Supercomputer Statistics & Top500 Project Start in 1993 Competition between Manufacturers, Countries and Sites My Supercomputer Favorite in the Top500 Lists The 30 th List as of November 2007  Performance Development and Projection Bell‘s Law Supercomputing, quo vadis? - in Jülich, Germany and Europe - in the Rest of the World Top500, quo vadis? Outline

31. JSC Jahresabschluss-Kolloquium 2007 page 31 Performance Development and Projection Performance Development Jack‘s Notebook

32. JSC Jahresabschluss-Kolloquium 2007 page 32 6-8 years 8-10 years Performance Development and Projection Performance Projection 1 Pflop/s Jack‘s Notebook

33. JSC Jahresabschluss-Kolloquium 2007 page 33 Mannheim Supercomputer Statistics & Top500 Project Start in 1993 Competition between Manufacturers, Countries and Sites My Supercomputer Favorite in the Top500 Lists The 30 th List as of November 2007 Performance Development and Projection  Bell‘s Law Supercomputing, quo vadis? - in Jülich, Germany and Europe - in the Rest of the World Top500, quo vadis? Outline

34. JSC Jahresabschluss-Kolloquium 2007 page 34 Bell‘s Law Bell's Law of Computer Class formation was discovered about 1972. It states that technology advances in semiconductors, storage, user interface and networking advance every decade enable a new, usually lower priced computing platform to form. Once formed, each class is maintained as a quite independent industry structure. This explains mainframes, minicomputers, workstations and Personal computers, the web, emerging web services, palm and mobile devices, and ubiquitous interconnected networks. We can expect home and body area networks to follow this path. From Gordon Bell (2007), http://research.microsoft.com/~GBell/Pubs.htm

35. JSC Jahresabschluss-Kolloquium 2007 page 35 Bell‘s Law Bell’s Law states, that: Important classes of computer architectures come in cycles of about 10 years. It takes about a decade for each phase : Early research Early adoption and maturation Prime usage Phase out past its prime Can we use Bell’s Law to classify computer architectures in the TOP500?

36. JSC Jahresabschluss-Kolloquium 2007 page 36 Bell‘s Law Gordon Bell (1972): 10 year cycles for computer classes Computer classes in HPC based on the TOP500: Data Parallel Systems: Vector (Cray Y-MP and X1, NEC SX, …) SIMD (CM-2, …) Custom Scalar Systems: MPP (Cray T3E and XT3, IBM SP, …) Scalar SMPs and Constellations (Cluster of big SMPs) Commodity Cluster: NOW, PC cluster, Blades, … Power-Efficient Systems (BG/L or BG/P as first example of low-power / embedded systems = potential new class ?) Tsubame with Clearspeed, Roadrunner with Cell ?

37. JSC Jahresabschluss-Kolloquium 2007 page 37 Bell‘s Law Computer Classes / Systems

38. JSC Jahresabschluss-Kolloquium 2007 page 38 Bell‘s Law Computer Classes - refined / Systems

39. JSC Jahresabschluss-Kolloquium 2007 page 39 Bell‘s Law ClassEarly Adoption starts: Prime Use starts: Past Prime Usage starts: Data Parallel Systems Mid 70’sMid 80’sMid 90’s Custom Scalar Systems Mid 80’sMid 90’sMid 2000’s Commodity Cluster Mid 90’sMid 2000’sMid 2010’s ??? BG/L or BG/PMid 2000’sMid 2010’s ???Mid 2020’s ??? HPC Computer Classes

40. JSC Jahresabschluss-Kolloquium 2007 page 40 Mannheim Supercomputer Statistics & Top500 Project Start in 1993 Competition between Manufacturers, Countries and Sites My Supercomputer Favorite in the Top500 Lists The 30 th List as of November 2007 Performance Development and Projection Bell‘s Law  Supercomputing, quo vadis ? - in Jülich, Germany and Europe - in the Rest of the World Top500, quo vadis? Outline

41. page 41 Supercomputing in Jülich und dem Rest der Welt Hans Werner Meuer ehem. Mitarbeiter des ZAM von 1962-73 Kolloquium zur Verabschiedung von Prof. Dr. Friedel Hoßfeld Forschungszentrum Jülich, 11. Juli 2002

42. 1997 0.18 * 10 12 Flop/s Cray T3E im FZJ 512 Prozessoren # 5 in TOP500 Liste 06/97

43. 1997-2002 35.86 * 10 12 Flop/s Earth Simulator Japan 5120 NEC Vektor- Prozessoren von da an ging‘s bergab… …mit dem Standing des FZJ in den TOP500!

44. JSC Jahresabschluss-Kolloquium 2007 page 44 Supercomputing, quo vadis? FZ Jülich Position in Top500

45. JSC Jahresabschluss-Kolloquium 2007 page 45 Supercomputing, quo vadis? European Countries / Systems

46. JSC Jahresabschluss-Kolloquium 2007 page 46 Supercomputing, quo vadis? Countries/Systems (November 2007)

47. JSC Jahresabschluss-Kolloquium 2007 page 47 Supercomputing, quo vadis? Countries/Performance (November 2007)

48. JSC Jahresabschluss-Kolloquium 2007 page 48 Supercomputing, quo vadis? TOP50 Countries/Systems (November 2007)

49. JSC Jahresabschluss-Kolloquium 2007 page 49 ISC’08 in Dresden June 17–20 2008 Conference Program – Thursday, June 19, 2008 Chair: Dr. Francesc Subirada, Barcelona Supercomputing Center, Spain  PRACE – Partnership for Advanced Computing in Europe Prof. Dr. Achim Bachem, Research Centre Jülich, Germany  The European PROSPECT Project Prof. Dr. Dr. Thomas Lippert, Jülich Supercomputing Centre (JSC), Germany Giovanni Trezza, Quadrics, Italy  The European TALOS Project Claude Camozzi, Bull, France Pierre Leca, CEA, France IV. European High End Activities 9:00 am – 10:30 am

50. JSC Jahresabschluss-Kolloquium 2007 page 50 Supercomputing, quo vadis? Continents/Systems

51. JSC Jahresabschluss-Kolloquium 2007 page 51 Supercomputing, quo vadis? Asian Countries / Systems

52. JSC Jahresabschluss-Kolloquium 2007 page 52 Supercomputing, quo vadis? Producing Regions / Systems

53. JSC Jahresabschluss-Kolloquium 2007 page 53 Mannheim Supercomputer Statistics & Top500 Project Start in 1993 Competition between Manufacturers, Countries and Sites My Supercomputer Favorite in the Top500 Lists The 30 th List as of November 2007 Performance Development and Projection Bell‘s Law Supercomputing, quo vadis? - in Jülich, Germany and Europe - in the Rest of the World  Top500, quo vadis? Outline

54. JSC Jahresabschluss-Kolloquium 2007 page 54 www.top500.org Top500, quo vadis?

55. JSC Jahresabschluss-Kolloquium 2007 page 55 Top500, quo vadis? TOP500 proofed itself by correcting the Mannheim Supercomputer Statistics. It is simplistic, but (or because of it) it gets trends right. It does not easily allow to track market size It’s inventory based. This smoothes seasonal fluctuation. Turn-over very high, so it still reflects recent developments. Summary after Fifteen Years of Experience Replacement Rate

56. JSC Jahresabschluss-Kolloquium 2007 page 56 Motivation for Additional Benchmarks Clearly need something more than Linpack HPC Challenge Benchmark and others Cons Emphasizes only “peak” CPU speed and number of CPUs Does not stress local bandwidth Does not stress the network No single number can reflect overall performance Top500, quo vadis? Linpack Benchmark Pros One number Simple to define and rank Allows problem size to change with machine and over time Allowing Competitions

57. JSC Jahresabschluss-Kolloquium 2007 page 57 Top500, quo vadis? Presented at ISC’06 June 27-30 2006

58. JSC Jahresabschluss-Kolloquium 2007 page 58 Top500, quo vadis?

59. JSC Jahresabschluss-Kolloquium 2007 page 59 Top500, quo vadis? http://www.green500.org Green500 Rank MFLOPS/WSiteComputer Total Power (kW) TOP500 Rank 1357.23 Science and Technology Facilities Council - Daresbury Laboratory Blue Gene/P Solution 31.10121 2352.25 Max-Planck-Gesellschaft MPI/IPP Blue Gene/P Solution 62.2040 3346.95IBM - Rochester Blue Gene/P Solution 124.4024 4336.21 Forschungszentrum Juelich (FZJ) Blue Gene/P Solution 497.602 5310.93 Oak Ridge National Laboratory Blue Gene/P Solution 70.4741 6210.56Harvard University eServer Blue Gene Solution 44.80170 7210.56 High Energy Accelerator Research Organization /KEK eServer Blue Gene Solution 44.80171 8210.56 IBM - Almaden Research Center eServer Blue Gene Solution 44.80172 9210.56IBM Research eServer Blue Gene Solution 44.80173 10210.56IBM Thomas J. Watson Research Center eServer Blue Gene Solution 44.80174

60. JSC Jahresabschluss-Kolloquium 2007 page 60 Vielen Dank für Ihr Aufmerksamkeit! Die Folien sind hier erhältlich: http://www.top500.org/slides/meuer