page 1 Supercomputing, quo Vadis? Erfahrungen aus 15 Jahren TOP500 und 30 Listen Hans Werner Meuer Universität Mannheim & Prometeus GmbH Jahresabschluss-Kolloquium Jülich Supercomputing Centre (JSC) Mittwoch, 19. Dezember 2007

JSC Jahresabschluss-Kolloquium 2007 page 2 Herzlichen Glückwunsch

JSC Jahresabschluss-Kolloquium 2007 page 3 30th List: The TOP10 ManufacturerComputer Rmax [TF/s] Installation SiteCountryYear#Cores 1IBM BlueGene/L eServer Blue Gene 478.2DOE/NNSA/LLNLUSA ,992 2IBM JUGENE BlueGene/P Solution Forschungszentrum Juelich Germany200765,536 3SGI SGI Altix ICE New Mexico Computing Applications Center USA200714,336 4HP Cluster Platform 3000 BL460c Computational Research Laboratories, TATA SONS India200714,240 5HP Cluster Platform 3000 BL460c Swedish Government Agency Sweden200713,728 6Sandia/Cray Red Storm Cray XT DOE/NNSA/SandiaUSA200626,569 7Cray Jaguar Cray XT3/XT DOE/ORNLUSA200723,016 8IBM BGW eServer Blue Gene 91.29IBM Thomas WatsonUSA200540,960 9Cray Franklin Cray XT NERSC/LBNLUSA200719,320 10IBM New York Blue eServer Blue Gene 82.16Stony Brook/BNLUSA200736,864

JSC Jahresabschluss-Kolloquium 2007 page 4 Mannheim Supercomputer Statistics & Top500 Project Start in 1993 Competition between Manufacturers, Countries and Sites My Supercomputer Favorite in the Top500 Lists The 30 th List as of November 2007 Performance Development and Projection Bell‘s Law Supercomputing, quo vadis? - in Jülich, Germany and Europe - in the Rest of the World Top500, quo vadis? Outline

JSC Jahresabschluss-Kolloquium 2007 page 5 Mannheim Supercomputer Statistics & Top500 Project Start in 1993 Mannheim Supercomputer Statistics 1986 – 1992 Presented at ISC’86 – ISC’92/Mannheim Supercomputer Seminars Counting of Supercomputers in the World 530 systems in the year 1992

JSC Jahresabschluss-Kolloquium 2007 page 6 Manufacturer Share 0% 20% 40% 60% 80% 100% NEC Hitachi Fujitsu CDC Cray Mannheim Supercomputer Statistics & Top500 Project Start in 1993

JSC Jahresabschluss-Kolloquium 2007 page 7 Countries Share 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Others Others Europe Germany France UK Japan USA Mannheim Supercomputer Statistics & Top500 Project Start in 1993

JSC Jahresabschluss-Kolloquium 2007 page 8 Mannheim Supercomputer Statistics & Top500 Project Start in 1993 Deficits of the Mannheim Supercomputer Statistics Not very reliable Vector Supercomputers only – „numerical“/ mainly „scientific and engineering“ applications Definition of „Supercomputer“ necessary

JSC Jahresabschluss-Kolloquium 2007 page 9 Mannheim Supercomputer Statistics & Top500 Project Start in 1993 Top500 Procedure Listing the 500 most powerful computers in the world Yardstick: Rmax of Linpack Solve Ax=b, dense problem, matrix is random Update twice a year: ISC’xy in June in Germany SC’xy in November in the U.S. All information available from the TOP500 webserver at:

JSC Jahresabschluss-Kolloquium 2007 page 10 Mannheim Supercomputer Statistics & Top500 Project Start in 1993 Top500 Authors Project was started in spring 1993 by: Hans W. MeuerErich Strohmaier Jack Dongarra Authors since: O6/1993 Horst Simon 11/2000

JSC Jahresabschluss-Kolloquium 2007 page 11 „CRPC researcher Jack Dongarra of the University of Tennessee and Oak Ridge National Laboratory is one of three renowned computer scientists who assemble the legendary TOP500 List of the world's fastest supercomputers. Released twice a year since 1993, the list features sites with the most powerful computer systems, determined with information from a questionnaire sent to high-performance computer (HPC) experts, computational scientists, manufacturers, and the Internet community at large. Dongarra, Hans Meuer of the University of Mannheim, and Erich Strohmaier of the University of Tennessee released their June 1999 TOP500 List at the 14th Mannheim Supercomputing Conference and Seminar, held June in Mannheim, Germany.“ Source: The Center for Research on Parallel Computation at Rice University Dongarra's TOP500 List of World's Fastest Supercomputers Released at Mannheim Conference Mannheim Supercomputer Statistics & Top500 Project Start in 1993

JSC Jahresabschluss-Kolloquium 2007 page 12 Mannheim Supercomputer Statistics & Top500 Project Start in 1993  Competition between Manufacturers, Countries and Sites My Supercomputer Favorite in the Top500 Lists The 30 th List as of November 2007 Performance Development and Projection Bell‘s Law Supercomputing, quo vadis? - in Jülich, Germany and Europe - in the Rest of the World Top500, quo vadis? Outline

JSC Jahresabschluss-Kolloquium 2007 page 13 Top500 Status 1 st Top500 List in June 1993 at ISC’93 in Mannheim 29 th Top500 List on June 27, 2007 at ISC’07 in Dresden 30 th Top500 List on November 13, 2007 at SC07 in Reno 31 st Top500 List on June 18, 2008 at ISC’08 in Dresden 32 nd Top500 List on November 18, 2008 at SC08 in Austin 33 rd Top500 List on June 24, 2009 at ISC’09 in Hamburg Acknowledged by HPC-users, manufacturers and media Competition between Manufacturers, Countries and Sites

JSC Jahresabschluss-Kolloquium 2007 page 14 CountriesCountShare % USA % Japan % Germany % France % United Kingdom % Australia91.80 % Italy61.20 % Netherlands61.20 % Switzerland40.80 % Canada30.60 % Denmark30.60 % Korea30.60 % Others % Totals % 1 st List as of 06/ %500Totals %55Others 1.80 %9India 2.00 %10China 0.20 %1Korea 0.20 %1Denmark 1.00 %5Canada 1.40 %7Switzerland 1.20 %6Netherlands 1.20 %6Italy 0.20 %1Australia 9.60 %48United Kingdom 3.40 %17France 6.20 %31Germany 4.00 %20Japan 56.60%283USA Share %CountCountries 30 th List as of 11/2007 Competition between Manufacturers, Countries and Sites

JSC Jahresabschluss-Kolloquium 2007 page 15 Competition between Manufacturers, Countries and Sites Countries/Systems

JSC Jahresabschluss-Kolloquium 2007 page 16 ManufacturersCountShare % Cray Research % Fujitsu % Thinking Machines % Intel % Convex % NEC % Kendall Square Research % MasPar % Meiko91.80 % Hitachi61.20 % Parsytec30.60 % nCube30.60 % Totals % 1 st List as of 06/ %35Others 100 %500Totals 4.80 %24Dell 4.40 %22SGI %232IBM ——nCube ——Parsytec 0.20 %1Hitachi/Fujitsu ——Meiko ——MasPar —— Kendall Square Research 0.40 %2NEC %166Hewlett-Packard 0.20 %1Intel ——Thinking Machines 0.60 %3Fujitsu 2.80 %14Cray Inc. Share %CountManufacturers 30 th List as of 11/2007 Competition between Manufacturers, Countries and Sites

JSC Jahresabschluss-Kolloquium 2007 page 17 Competition between Manufacturers, Countries and Sites Manufacturer/Systems

JSC Jahresabschluss-Kolloquium 2007 page 18 Top Sites through 30 Lists Competition between Manufacturers, Countries and Sites RankSiteCountry% over time 1Lawrence Livermore National LaboratoryUnited States5.39 2Sandia National LaboratoriesUnited States3.70 3Los Alamos National LaboratoryUnited States3.41 4GovernmentUnited States3.34 5The Earth Simulator CenterJapan1.99 6National Aerospace Laboratory of JapanJapan1.70 7Oak Ridge National LaboratoryUnited States1.39 8NCSAUnited States1.31 9NASA/Ames Research Center/NASUnited States University of TokyoJapan NERSC/LBNLUnited States Pittsburgh Supercomputing CenterUnited States Semiconductor Company (C)United States Naval Oceanographic Office (NAVOCEANO)United States ECMWFUnited Kingdom ERDC MSRCUnited States IBM Thomas J. Watson Research CenterUnited States Forschungszentrum Juelich (FZJ)Germany Japan Atomic Energy Research InstituteJapan Minnesota Supercomputer CenterUnited States0.74

JSC Jahresabschluss-Kolloquium 2007 page 19 Mannheim Supercomputer Statistics & Top500 Project Start in 1993 Competition between Manufacturers, Countries and Sites  My Supercomputer Favorite in the Top500 Lists The 30 th List as of November 2007 Performance Development and Projection Bell‘s Law Supercomputing, quo vadis? - in Jülich, Germany and Europe - in the Rest of the World Top500, quo vadis? Outline

JSC Jahresabschluss-Kolloquium 2007 page 20 ManufacturerComputerRmax [GF/s] Installation SiteCountryYear#Proc 1IntelASCI Red1068Sandia National Laboratories USA HitachiCP-PACS/ Center for Computational Science Japan FujitsuNumerical Wind Tunnel 229National Aerospace Laboratory Japan HitachiSR2201/ University of TokyoJapan CrayT3E176Forschungszentrum Jülich Germany CrayT3E176GovernmentUSA CrayT3E176Max-Planck-Gesell. MPI/IPP Germany CrayT3E176NASA/Goddard Space Flight Center USA CrayT3E176Pittsburgh Supercomp. Center USA CrayT3E176University StuttgartGermany My Supercomputer Favorite in the Top500 Lists Top10 in the 06/1997 list

JSC Jahresabschluss-Kolloquium 2007 page 21 My Supercomputer Favorite in the Top500 Lists Performance [GFlop/s] N=1 N=500 Sum 1 TFlop/s 1 PFlop/s Intel ASCI Red Jun 05

JSC Jahresabschluss-Kolloquium 2007 page 22 Number of Teraflop/s Systems in the Top Jun-97Jun-98Jun-99Jun-00Jun-01Jun-02Jun-03Jun-04Jun-05 My Supercomputer Favorite in the Top500 Lists

JSC Jahresabschluss-Kolloquium 2007 page 23 My Supercomputer Favorite in the Top500 Lists

JSC Jahresabschluss-Kolloquium 2007 page 24 Mannheim Supercomputer Statistics & Top500 Project Start in 1993 Competition between Manufacturers, Countries and Sites My Supercomputer Favorite in the Top500 Lists  The 30 th List as of November 2007 Performance Development and Projection Bell‘s Law Supercomputing, quo vadis? - in Jülich, Germany and Europe - in the Rest of the World Top500, quo vadis? Outline

JSC Jahresabschluss-Kolloquium 2007 page 25 The 30 th List as of November 2007 Processor Architecture/Systems

JSC Jahresabschluss-Kolloquium 2007 page 26 The 30 th List as of November 2007 Operating Systems/Systems

JSC Jahresabschluss-Kolloquium 2007 page 27 The 30 th List as of November 2007 Processor Generation/Systems (November 2007)

JSC Jahresabschluss-Kolloquium 2007 page 28 The 30 th List as of November 2007 Interconnect Family/Systems

JSC Jahresabschluss-Kolloquium 2007 page 29 The 30 th List as of November 2007 Architectures / Systems

JSC Jahresabschluss-Kolloquium 2007 page 30 Mannheim Supercomputer Statistics & Top500 Project Start in 1993 Competition between Manufacturers, Countries and Sites My Supercomputer Favorite in the Top500 Lists The 30 th List as of November 2007  Performance Development and Projection Bell‘s Law Supercomputing, quo vadis? - in Jülich, Germany and Europe - in the Rest of the World Top500, quo vadis? Outline

JSC Jahresabschluss-Kolloquium 2007 page 31 Performance Development and Projection Performance Development Jack‘s Notebook

JSC Jahresabschluss-Kolloquium 2007 page years 8-10 years Performance Development and Projection Performance Projection 1 Pflop/s Jack‘s Notebook

JSC Jahresabschluss-Kolloquium 2007 page 33 Mannheim Supercomputer Statistics & Top500 Project Start in 1993 Competition between Manufacturers, Countries and Sites My Supercomputer Favorite in the Top500 Lists The 30 th List as of November 2007 Performance Development and Projection  Bell‘s Law Supercomputing, quo vadis? - in Jülich, Germany and Europe - in the Rest of the World Top500, quo vadis? Outline

JSC Jahresabschluss-Kolloquium 2007 page 34 Bell‘s Law Bell's Law of Computer Class formation was discovered about It states that technology advances in semiconductors, storage, user interface and networking advance every decade enable a new, usually lower priced computing platform to form. Once formed, each class is maintained as a quite independent industry structure. This explains mainframes, minicomputers, workstations and Personal computers, the web, emerging web services, palm and mobile devices, and ubiquitous interconnected networks. We can expect home and body area networks to follow this path. From Gordon Bell (2007),

JSC Jahresabschluss-Kolloquium 2007 page 35 Bell‘s Law Bell’s Law states, that: Important classes of computer architectures come in cycles of about 10 years. It takes about a decade for each phase : Early research Early adoption and maturation Prime usage Phase out past its prime Can we use Bell’s Law to classify computer architectures in the TOP500?

JSC Jahresabschluss-Kolloquium 2007 page 36 Bell‘s Law Gordon Bell (1972): 10 year cycles for computer classes Computer classes in HPC based on the TOP500: Data Parallel Systems: Vector (Cray Y-MP and X1, NEC SX, …) SIMD (CM-2, …) Custom Scalar Systems: MPP (Cray T3E and XT3, IBM SP, …) Scalar SMPs and Constellations (Cluster of big SMPs) Commodity Cluster: NOW, PC cluster, Blades, … Power-Efficient Systems (BG/L or BG/P as first example of low-power / embedded systems = potential new class ?) Tsubame with Clearspeed, Roadrunner with Cell ?

JSC Jahresabschluss-Kolloquium 2007 page 37 Bell‘s Law Computer Classes / Systems

JSC Jahresabschluss-Kolloquium 2007 page 38 Bell‘s Law Computer Classes - refined / Systems

JSC Jahresabschluss-Kolloquium 2007 page 39 Bell‘s Law ClassEarly Adoption starts: Prime Use starts: Past Prime Usage starts: Data Parallel Systems Mid 70’sMid 80’sMid 90’s Custom Scalar Systems Mid 80’sMid 90’sMid 2000’s Commodity Cluster Mid 90’sMid 2000’sMid 2010’s ??? BG/L or BG/PMid 2000’sMid 2010’s ???Mid 2020’s ??? HPC Computer Classes

JSC Jahresabschluss-Kolloquium 2007 page 40 Mannheim Supercomputer Statistics & Top500 Project Start in 1993 Competition between Manufacturers, Countries and Sites My Supercomputer Favorite in the Top500 Lists The 30 th List as of November 2007 Performance Development and Projection Bell‘s Law  Supercomputing, quo vadis ? - in Jülich, Germany and Europe - in the Rest of the World Top500, quo vadis? Outline

page 41 Supercomputing in Jülich und dem Rest der Welt Hans Werner Meuer ehem. Mitarbeiter des ZAM von Kolloquium zur Verabschiedung von Prof. Dr. Friedel Hoßfeld Forschungszentrum Jülich, 11. Juli 2002

* Flop/s Cray T3E im FZJ 512 Prozessoren # 5 in TOP500 Liste 06/97

* Flop/s Earth Simulator Japan 5120 NEC Vektor- Prozessoren von da an ging‘s bergab… …mit dem Standing des FZJ in den TOP500!

JSC Jahresabschluss-Kolloquium 2007 page 44 Supercomputing, quo vadis? FZ Jülich Position in Top500

JSC Jahresabschluss-Kolloquium 2007 page 45 Supercomputing, quo vadis? European Countries / Systems

JSC Jahresabschluss-Kolloquium 2007 page 46 Supercomputing, quo vadis? Countries/Systems (November 2007)

JSC Jahresabschluss-Kolloquium 2007 page 47 Supercomputing, quo vadis? Countries/Performance (November 2007)

JSC Jahresabschluss-Kolloquium 2007 page 48 Supercomputing, quo vadis? TOP50 Countries/Systems (November 2007)

JSC Jahresabschluss-Kolloquium 2007 page 49 ISC’08 in Dresden June 17– Conference Program – Thursday, June 19, 2008 Chair: Dr. Francesc Subirada, Barcelona Supercomputing Center, Spain  PRACE – Partnership for Advanced Computing in Europe Prof. Dr. Achim Bachem, Research Centre Jülich, Germany  The European PROSPECT Project Prof. Dr. Dr. Thomas Lippert, Jülich Supercomputing Centre (JSC), Germany Giovanni Trezza, Quadrics, Italy  The European TALOS Project Claude Camozzi, Bull, France Pierre Leca, CEA, France IV. European High End Activities 9:00 am – 10:30 am

JSC Jahresabschluss-Kolloquium 2007 page 50 Supercomputing, quo vadis? Continents/Systems

JSC Jahresabschluss-Kolloquium 2007 page 51 Supercomputing, quo vadis? Asian Countries / Systems

JSC Jahresabschluss-Kolloquium 2007 page 52 Supercomputing, quo vadis? Producing Regions / Systems

JSC Jahresabschluss-Kolloquium 2007 page 53 Mannheim Supercomputer Statistics & Top500 Project Start in 1993 Competition between Manufacturers, Countries and Sites My Supercomputer Favorite in the Top500 Lists The 30 th List as of November 2007 Performance Development and Projection Bell‘s Law Supercomputing, quo vadis? - in Jülich, Germany and Europe - in the Rest of the World  Top500, quo vadis? Outline

JSC Jahresabschluss-Kolloquium 2007 page 54 Top500, quo vadis?

JSC Jahresabschluss-Kolloquium 2007 page 55 Top500, quo vadis? TOP500 proofed itself by correcting the Mannheim Supercomputer Statistics. It is simplistic, but (or because of it) it gets trends right. It does not easily allow to track market size It’s inventory based. This smoothes seasonal fluctuation. Turn-over very high, so it still reflects recent developments. Summary after Fifteen Years of Experience Replacement Rate

JSC Jahresabschluss-Kolloquium 2007 page 56 Motivation for Additional Benchmarks Clearly need something more than Linpack HPC Challenge Benchmark and others Cons Emphasizes only “peak” CPU speed and number of CPUs Does not stress local bandwidth Does not stress the network No single number can reflect overall performance Top500, quo vadis? Linpack Benchmark Pros One number Simple to define and rank Allows problem size to change with machine and over time Allowing Competitions

JSC Jahresabschluss-Kolloquium 2007 page 57 Top500, quo vadis? Presented at ISC’06 June

JSC Jahresabschluss-Kolloquium 2007 page 58 Top500, quo vadis?

JSC Jahresabschluss-Kolloquium 2007 page 59 Top500, quo vadis? Green500 Rank MFLOPS/WSiteComputer Total Power (kW) TOP500 Rank Science and Technology Facilities Council - Daresbury Laboratory Blue Gene/P Solution Max-Planck-Gesellschaft MPI/IPP Blue Gene/P Solution IBM - Rochester Blue Gene/P Solution Forschungszentrum Juelich (FZJ) Blue Gene/P Solution Oak Ridge National Laboratory Blue Gene/P Solution Harvard University eServer Blue Gene Solution High Energy Accelerator Research Organization /KEK eServer Blue Gene Solution IBM - Almaden Research Center eServer Blue Gene Solution IBM Research eServer Blue Gene Solution IBM Thomas J. Watson Research Center eServer Blue Gene Solution

JSC Jahresabschluss-Kolloquium 2007 page 60 Vielen Dank für Ihr Aufmerksamkeit! Die Folien sind hier erhältlich: