1. Barcelona, 16 de Noviembre de 2004 Prof. Mateo Valero BSC Director Barcelona Supercomputing Center Centro Nacional de Supercomputación

2. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 TheoryExperiment Computing & Simulation High Performance Computing Aircraft, Automobile Design Fusion Reactor, Accelerator Design, Material Science, Astrophysics Climate and Weather Modeling

3. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 What drives HPC ? --- “The Need for Speed...” Computational Needs of Technical, Scientific, Digital Media and Business Applications Approach or Exceed the Petaflops/s Range CFD Wing Simulation 512x64x256 Grid (8.3 x10e6 mesh points) 5000 FLOPS per mesh point, 5000 time steps/cycles 2.15x10e14 FLOPS CFD Full Plane Simulation 512x64x256 Grid (3.5 x10e17 mesh points) 5000 FLOPS per mesh point, 5000 time steps/cycles 8.7x10e24 FLOPS Source: A. Jameson, et al Materials Science Magnetic Material: Current: 2000 atoms; 2.64 TF/s, 512 GB Future: HDD Simulation - 30 TF/s, 2 TB Electronic Structures: Current: 300 atoms; 0.5 TF/s, 100 GB Future: 3000 atoms; 50 TF/s, 2TB Source: D. Balley, NERSC Digital Movies and Special Effects ~1e14 FLOPs per frame 50 frames/sec 90 minute movie - 2.7e19 FLOPs ~ 150 days on 2000 1GFLOP/s CPUs Source: Pixar Spare Parts Inventory Planning Modelling the optimized deployment of 10000 part numbers across 100 part depots and requries: - 2x10e14 FLOP/s (12 hours on 10, 650 MHz CPUs) - 2.4 PetaFlop/s sust. performance (1 hour turn-around time) Industry trend to rapid, frequent modeling for timely business decision support driver higher sustained performance Source: B. Dietrich, IBM

4. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 Technological Achievements Transistor (Bell Labs, 1947) DEC PDP-1 (1957) IBM 7090 (1960) Integrated circuit (1958) Kilburn, TI/Fairchaild IBM System 360 (1965) DEC PDP-8 (1965) Microprocessors (1971) Intel 4004

5. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 In 50++ years... Eniac, 1946... 1900 vacuum tubes Power5, 2004 First dual core SMT processor 276M transistores 389 mm 2 1,8 GHz 1,9 MB L2 cache

6. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 Technology Directions: SIA Roadmap Year199920022005200820112014 Feature size (nm)180130100705035 Logic trans/cm 2 6.2M18M39M84M180M390M Cost/trans (mc)1.735.580.255.110.049.022 #pads/chip186725533492477665328935 Clock (MHz)12502100350060001000016900 Chip size (mm 2 )340430530620750900 Wiring levels6-777-88-9910 Power supply (V)1.81.51.20.90.60.5 High-perf pow (W) 90130160170175183 Battery pow (W)1.422.42.83.23.7

7. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 Applications for Supercomputers Aircraft/car simulations Life Science (Proteins, Human Organs,…) Atmosphere Stars Nanomaterials Drugs Regional/Global Climate/Weather/Pollution High Energy Physics Combustion Image Processing

8. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 Suitable applications for massively parallel systems Source: Rick Stevens, Argonne National Lab and The University of Chicago

9. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 Throughput vs. Parallel programming Throughput Multiple, unrelated, instruction streams (programs) that execute concurrently on multiple processors Multiprogramming n tasks on p processors: each task receives p/n processors Parallel Programming Multiple related, interacting instructions (single program) that execute concurrently to increase the speed of a single program 1 task on p processors, each processor receives 1/p of the task: reduce response time

10. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 Distributed and Parallel Systems Distributed systems heterogenous Massively parallel systems homogenous SETI@hom e Entropia/UD Grid based Computing Google Network of ws Cluster w/ special interconnect Parallel Dist. memASCI Tflop/sBlue Gene ● Gather (unused) resources ● Steal cycles ● System SW manages resources ● System SW adds value ● 10% - 20% overhead is OK ● Resources drive applications ● Time to completion is not critical ● Time-shared ● SETI@home ≈ 400,000 machines Averaging 40 Tflop/s ● Bounded set of resources ● Apps grow to consume all cycles ● Application manages resources ● System SW gets in the way ● 5% overhead is maximum ● Apps drive purchase of equipment ● Real-time constraints ● Space-shared ● Blue Gene/L 32768 processors Averaging 70,72 Tflop/s

11. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 Google query attributes 150M queries/day (2000/second) 3B document in the index Data centers 15,000 Linux systems in 6 data centers 15 TFlop/s and 1000TB total capability 40-80 1U/2U servers/cabinet 100 MB Ethernet switches with Gigabit Ethernet uplink growth from 4,000 systems (June 2000) 18M queries then Performance and operation simple reissue of failed commands to new servers no performance debugging problems are not reproducible Source: Monika Henzinger, Google

12. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 Life Science future projects In silico drug screening Clustering of Expressed Sequence Tags In silico Structural Genomics First ab-initio Calculations of the Electronic Structure of the Smallest Living Organism Mixed QM/MM Simulations of the Five Most Important Enzymatic Reactions Virtual Cell Project Digital human and virtual chirurgy Ab-initio Food Science Protein Folding Dynamics

13. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 The structure and dynamics of protein folding determine their biological properties Some illness related to misfoldings are Alzheimer’s Disease Mad Cow Disease Cystic Fibrosis Supercomputing will enable the in- silico simulation of protein folding, contributing to novel drug development techniques and new medical treatments Better understanding of biological process

14. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 DescriptionCountComment Atoms≈ 32000260 amino acid protein + water Force evaluation / time step10 9 Pairwise atom-atom interaction FLOPs / force evaluation150Typical molecular dynamics FLOPSs / time step1.5 x 10 11 Each time step≈ 10 -15 s0.5 - 5 femto second Total simulation time single step10 -3 sProtein folds in ≈1 mili second Total time steps2 x 10 11 FLOPs / simulation3 x 10 22 Total FLOPs to fold a protein Execution time3 x 10 7 s1 year Required FLOPS≈ 10 15 1 Petaflop Protein Folding Calculation

15. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 Personalized medicine requires massive supercomputing resources Each individual has a different genetic sequence. Its understanding would help to identify specific risks Sequencing the DNA of the human population implies the analysis of 1.8 million TB/s (300 TB/person * 6 billion person) The comparison of the genetic profiles of an individual against a central database would allow the development of life-long rational healthcare with food and drugs adequate to the individual genetic characteristics

16. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 Supercomputing facilitates accelerated and significant progress in the earth and material sciences Biogeochemical cycles Biodiversity and ecosystems in equilibrium Climatic variations Hydrological predictions Contagious diseases and environmental impact Rational use of resources Reinventing materials and their applications Supercomputing will enable to simulate the environmental and social impact of agricultural, industrial and urbanistic policies, as well as the impact of external factors (desertification, catastrophic events) in real time Protecting and improving our environment

17. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 CEPBA: European Center for Parallelism of Barcelona Dept.CER DACCEPBA ● October 1991 ● R&D on parallelism ● Training ● Technology transfer ● European Context ● HPC experience RME, LSI, FEN, FA ● Computing needs Sponsored by:

18. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 CEPBA: joint ventures Centre de Computació i Comunicacions de Catalunya ● October 1995 ● Coordination between CEPBA IBM Research Institute ● Starting activities November 2000 ● Research Cooperation in Deep Computing ● Support research on other fields of Science

19. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 CEPBA: Service Manuf.ModelProcessor Memory Disc Peak Performance Service IBMSP3+p360 128 Power3 + 36 Power4 64+18 GB 1,8 TB 336 Gflop/s 10/01 12/02 Compaq Alpha Server GS-160 16 Alpha 21264 8 GB 108 GB 23,3 Gflop/s9/00 ParsytecCCi-8D 16 Pentium II 1 GB 30 GB 3,2 Gflop/s4/98 Silicon Graphics Origin 2000 64 MIPS R10000 8 GB 360 GB 25 Gflop/s1/97 Digital Alpha Server 8400 12 Alpha 21164 2 GB 32 GB 10,5 Gflop/s12/96 Silicon Graphics Power Challenge 12+12 R8000/R10000 4 GB 50 GB 7/96 - 7/97 Thinking Machine CM-2 2048 1 bit 256 MB 640 Mflops4/92 - 2/98 ConvexC34808 1 GB 16 GB 0,4 Gflop/s10/91 - 1/98 ParsysSN 1000 32 T800 128 MB 2 GB 64 Mflops5/90 - 5/95

20. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 CEPBA: European Mobility Programs Joint CEPBA - CESCA projects Stays and access to resources ProjectPeriodFundingVisitors HCM1993-1997950112 PECO1995-199616017 TMR1996-2000935133 IHP2000-2003700142 Total2745404

21. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 CEPBA: R&D projects

22. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 CEPBA: R&D projects

23. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 CEPBA: Management of Technology Transfer

24. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 The Consortium Consortium includes Spanish Government (MEC) Catalonian Government (DURSI) Technical University of Catalunya To start operations on January, 2005 Location c/ Jordi Girona 31 08034 Barcelona

25. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 Significant contribution to advancement of Science in Spain, enabling supercomputing capacity, scientific-technical synergies, and cost rationalization thanks to economies of scale Powerful tool to assist research and development centers, public and private, generating impulses for a new technological environment Motivation

26. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 Mission ” Investigate, develop and manage technology to facilitate the advancement of science ”

27. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 Objectives Research in Supercomputing and Computer Architecture Collaborate in R&D e-Science projects with prestigious scientific teams Manage BSC supercomputers to accelerate relevant contributions to research areas where intensive computing is an enabling technology

28. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 Structure

29. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 IT research and development projects Continuation of CEPBA (European Center for Parallelism in Barcelona) research lines Deep Computing Performance Tools Parallel programing Grid Code Optimization Computer Architecture Vector processors Network processors

30. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 e-Science projects R&D collaborations Computational Biology Computational Chemistry Computational Physics Information based Medicine

31. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 Management projects Supercomputer Management System Administration Users support Business Development External Relations Promotion Technology Transfer Education Administration Accounting and Finances Human Resources

32. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 MareNostrum PowerPC 970 FX processors (dual processors) 4GB ECC 333 DDR memory per node 3 networks Myrinet Gigabit 10/00 Ethernet Diskless network support Linux cluster

33. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 27 Compute Racks (RC01-RC27) 162 BC chassis w/OPM and gigabit ether switch 2268 JS20+ nodes w/myrinet daughter card 7 Storage Server Racks (RS01-RS07) 40 p615 storage servers 6/rack 20 FastT 100 3/rack 20 EXP100 3/rack 4 Myrinet Racks (RM01-RM04) 10 clos256+256 myrinet switches 2 Myrinet spines 1280s 1 Gigabit Network Racks 1 Force10 E600 for Gb network 4 Cisco 3550 48-port for 10/100 network 1 Operations Rack (RH01) 7316-TF3 display 2 p615 mgmt nodes 2 HMC model 7315-CR2 3 Remote Async Nodes 3 Cisco 3550 1 BC chassis (BCIO) MareNostrum: Overall system description

34. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 Processor: PowerPC 970FX

35. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 JS20 Processor Blade 2-way 2.2 GHz Power PC 970 SMP 4GB memory (512KB L2 cache) Local IDE drive (40 GB) 2x1Gb Ethernet on board Myrinet daughter card Blade Center 14 blades per chassis (7U) 28 processors 56GB memory Gigabit ethernet switch Blades, blade center and blade center racks 6 chassis in a rack (42U) 168 processors 336GB memory

36. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 Blade centers Myrinet racks Storage servers Operations rack Gigabit switch 10/100 switches MareNostrum: Floor plan with cabling 27 racks + 1 BC chassis 4564 processors 9TB memory

37. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 Blade centers Myrinet racks Storage servers Operations rack Gigabit switch 10/100 switches

38. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 256 blades connected to 1 clos 256+256 Myrinet 1280 blades connected to 5 clos 256+256 Myrinet and 1 spine 1280 2282 blades connected to 10 clos 256+256 Myrinet and 2 spine 1280 20 x 7TB storage nodes Management rack, Force 10 Gigabit, 10/100 Cisco switches

39. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 Blade center racks 6 Blade Centers per rack 27 racks + 1 Blade Center Cabling per rack 84 fiber cables to myrinet switch 6 Gb to Force10 E600 6 10/100 cat5 to Cisco Blade Center

40. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 Myrinet racks 10 Clos 256x256 switches Interconnect up to 256 Blades Connect to Spine (64 ports) 2 Spine 1280 Interconnect up to 10 Clos 256x256 switches Monitoring using 10/100 connection

41. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 Spine 1280Clos 256x256 256 Blades or Storage servers 64 to Spine 320 from Clos Myrinet racks

42. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 Interconnection of Blade Centers Used for system boot of every blade center 212 internal network cables 170 for blades 42 for p615 76 connection available to external connection Gb Subsystem: Force 10 E600

43. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 Storage nodes Total of 20 storage nodes, 20 x 7 TBytes Each storage node 2xP615 FastT100 EXP100 Cabling per node 2 Myrinet 2 Gb to Force10 E600 2 10/100 cat5 to Cisco 1 Serial P615 FastT100 EXP100 P615 FastT100 EXP100 P615 FastT100 EXP100

44. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 Management rack Contents BCIO Display 2 HMC 2 x p615 3 x Cisco 3 x 16-port Remote Async Nodes

45. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 Scientific Applications Source: Bob Voigt, NSF

46. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004

47. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004

48. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004

49. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004

50. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 MareNostrum: number comparison Project Columbia BlueGene /L Earth Simulator MareNostru m Number of racks>6464040 Number of processors10240128k51204500 System peak performance65 TF360 TF41 TF40 TF Total system Memory16 TB10 TB9 TB Power< 1.6 MW5.1 MW600 KW Floor space400 m 2 3250 m 2 120 m 2 Bisectional bandwidth0.7 TB/s7.8 TB/s0.6 TB/s Number of cables5000832002600

51. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 Thank you ! Prof. Mateo Valero BSC Director

52. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004

53. BSC - Barcelona Supercomputer Center Barcelona, 16 de Noviembre de 2004 From July 7th to October 20th 1 image every hour on working days From November 27th to December 7th 1 image every 30 seconds