1. National Center for Supercomputing Applications University of Illinois at Urbana-Champaign Future of High Performance Computing Thom Dunning National Center for Supercomputing Applications

2. Outline of Presentation Directions in Computing Technology From uni-core to multi-core chips On to many-core chips From Terascale to Petascale Computing Science @ Petascale Blue Waters Petascale Computing System Path to Exascale Computing Issues for beyond petascale computing Take Home Lessons Petascale Summer School 6-9 July 2010 Urbana, Illinois

3. A major shift is underway in computing technology with multicore and many-core chips Directions in Computing Technology

4. Directions in Computing Technology Increasing Performance of Microprocessors Frequency (MHz) “In the past, performance scaling in conventional single-core processors has been accomplished largely through increases in clock frequency (accounting for roughly 80 percent of the performance gains to date).” Platform 2015 S. Y. Borkar et al., 2006 Intel Corporation Intel Pentium Petascale Summer School 6-9 July 2010 Urbana, Illinois

5. Directions in Computing Technology Problem with Uni-core Microprocessors Decreasing Feature Size Increasing Chip Frequency Watts/cm 2 1 10 100 1000 1.5  1.0  0.7  0.5  0.35  0.25  0.18  0.13  0.1  0.07  i386 i486 Pentium Pentium Pro Pentium II Pentium III Hot Plate Nuclear Reactor Rocket Nozzle Pentium 4 (Prescott) Pentium 4 (Willamette) Petascale Summer School 6-9 July 2010 Urbana, Illinois

6. Directions in Computing Technology From Uni-core to Multi-core Processors Intel’s Nehalem Modular Up to 8 cores 3 levels of cache Integrated memory controller Multiple QuickPath Interconnects Petascale Summer School 6-9 July 2010 Urbana, Illinois

7. Directions in Computing Technology Switch to Multicore Chips Frequency (MHz) dual core quad core “For the next several years the only way to obtain significant increases in microprocessor performance will be through increasing use of parallelism: –8× in 2009-10, –16× in 2011-12, –and so on Petascale Summer School 6-9 July 2010 Urbana, Illinois

8. Directions in Computing Technology On to Many-core Chips Intel Teraflops Chip (80 cores) NVIDIA Fermi (512 cores) AMD Llano (4 x86 cores + 480 stream processors) Petascale Summer School 6-9 July 2010 Urbana, Illinois Intel Many Integrated Cores (>80 x86+ cores)

9. Directions in Computing Technology Recent Evolution of NVIDIA GPUs GPU:G80GT200Fermi Transistors681 million1,400 million3,000 million CUDA Cores128240512 DP Floating PointNone30 FMA/cycle256 FMA/cycle SP Floating Point128 MAD/cycle240 MAD/cycle512 FMA/cycle Shared Memory16 KB/SM 16 or 48 KB/SM L1 CacheNone 16 or 48 KB/SM L2 CacheNone 768 KB ECC MemoryNo Yes Address Width32-bit 64-bit Petascale Summer School 6-9 July 2010 Urbana, Illinois Peak DP performance = 256 FMA/cycle x 2 flops/FMA x 1.5 GHz = 768 GF Peak SP performance = 512 FMA/cycle x 2 flops/FMA x 1.5 GHz = 1,536 GF

10. Directions in Computing Technology Fermi Streaming Multiprocessor Architecture Streaming Multiprocessor 16 SMs per chip Each SM has: 32 CUDA cores Floating point and integer units for each core Fused multiply-add instruction 16 load-store units 4 special function units Transcendental functions (sin, cos, reciprocal, square root) Petascale Summer School 6-9 July 2010 Urbana, Illinois

11. Directions in Computing Technology AMD’s Fusion “Application Processing Unit” Heterogeneous Architecture x86 cores Streaming processors High Performance Interconnect High Performance Memory Controller Petascale Summer School 6-9 July 2010 Urbana, Illinois

12. A computing system for solving the most challenging compute-, memory- and data-intensive problems Blue Waters: From Terascale to Petascale Computing

13. Blue Waters NSF Track 1 Solicitation Petascale Summer School 6-9 July 2010 Urbana, Illinois “The petascale HPC environment will enable investigations of computationally challenging problems that require computing systems capable of delivering sustained performance approaching 10 15 floating point operations per second (petaflops) on real applications, that consume large amounts of memory, and/or that work with very large data sets.” Leadership-Class System Acquisition - Creating a Petascale Computing Environment for Science and Engineering NSF 06-573

14. Blue Waters Computational Science and Engineering Petascale Summer School 6-9 July 2010 Urbana, Illinois Molecular ScienceWeather & Climate Forecasting Earth ScienceAstronomy Health Petascale computing will enable advances in a broad range of science and engineering disciplines:

15. Blue Waters Desired Attributes of Petascale System Maximum Core Performance …to minimize number of cores needed for a given performance level, lessen impact of sections of code with limited scalability Low Latency, High Bandwidth Interconnect …to enable science and engineering applications to scale to tens to hundreds of thousands of cores Large, Fast Memories …to solve the most memory-intensive problems Large, Fast I/O System and Data Archive …to solve the most data-intensive problems Reliable Operation …to enable the solution of Grand Challenge problems Petascale Summer School 6-9 July 2010 Urbana, Illinois

16. Blue Waters Building Blue Waters Petascale Summer School 6-9 July 2010 Urbana, Illinois IH Server Node 8 QCM’s (256 cores) 8 TF (peak) 1 TB memory 4 TB/s memory bw 8 Hub chips Power supplies PCIe slots Fully water cooled Quad-chip Module 4 Power7 chips 128 GB memory 512 GB/s memory bw 1 TF (peak) Hub Chip 1,128 GB/s bw Blue Waters is built from components that can be used to build systems with a wide range of capabilities—from servers to beyond Blue Waters. Blue Waters will be the most powerful computer in the world for scientific research when it comes on line in Summer of 2011. Power7 Chip 8 cores, 32 threads L1, L2, L3 cache (32 MB) Up to 256 GF (peak) 128 Gb/s memory bw 45 nm technology Blue Waters ~10 PF Peak ~1 PF sustained >300,000 cores ~1.2 PB of memory >18 PB of disk storage 500 PB of archival storage ≥100 Gbps connectivity Blue Waters Building Block 32 IH server nodes 256 TF (peak) 32 TB memory 128 TB/s memory bw 4 Storage systems (>500 TB) 10 Tape drive connections

17. Blue Waters Comparison: Jaguar and Blue Waters Petascale Summer School 6-9 July 2010 Urbana, Illinois ORNLNCSA System AttributeJaguar (#1)Blue Waters Vendor (Model)Cray (XT5)IBM (PERCS) ProcessorAMD Opteron IBM Power7 Peak Performance (PF)2.3~10 Sustained Performance (PF)? ≳ 1 Number of Cores/Chip68 Number of Processor Cores224,256>300,000 Amount of Memory (TB)299~1,200 Amount of On-line Disk Storage (PB)5>18 Sustained Disk Transfer (TB/sec)0.24>1.5 Amount of Archival Storage (PB)20up to 500 ~ 4 1⅓ < 1½ 4 > 3 > 6 25

18. Blue Waters Project Critical Features of Blue Waters. I High Performance Compute Module SMP system Four Power7 chips Hub chip Performance: 1 TF Memory: 128 GB High Performance Interconnect High bandwidth, low latency Hub Chip/QCM: > 1 TB/sec/QCM Latency: ~ 1  sec Fully connected, two tier network Copper + optical links Petascale Summer School 6-9 July 2010 Urbana, Illinois

19. Blue Waters Project Critical Features of Blue Waters. II High Performance I/O and Data archive Systems Large storage subsystems On-line disks: > 18 PB (usable) Archival tapes: Up to 500 PB High sustained disk transfer rate: > 1.5 TB/sec (sustained) Fully integrated storage system: GPFS + HPSS General Hardware support for global shared memory Petascale Summer School 6-9 July 2010 Urbana, Illinois

20. Blue Waters National Petascale Computing Facility Petascale Summer School 6-9 July 2010 Urbana, Illinois Partners EYP MCF/ Gensler IBM Yahoo! Modern Data Center 90,000+ ft 2 total 30,000 ft 2 raised floor 20,000 ft 2 machine gallery Energy Efficiency LEED certified Gold (goal: Platinum) PUE = 1.1–1.2

21. Although an exascale computer is at least 10 years away, the issues being confronted will impact all systems beyond Blue Waters Path to Exascale Computing Petascale Summer School 6-9 July 2010 Urbana, Illinois

22. Blue Waters A Glimpse into the Future: Sequoia Petascale Summer School 6-9 July 2010 Urbana, Illinois NCSALLNL System AttributeBlue WatersSequoia Vendor (Model)IBM (PERCS)IBM BG/Q ProcessorIBM Power7IBM PowerPC Peak Performance (PF)~10~20 Sustained Performance (PF) ≳ 1? Number of Cores/Chip816 Number of Processor Cores>300,000~1,600,000 Amount of Memory (TB)~1,200~1,600 Amount of On-line Disk Storage (PB)>18~50 Sustained Disk Transfer (TB/sec)>1.50.5–1.0

23. Path from Petascale to Exascale Levels of concurrency Cores: 100s of thousands ➙ 100s of millions Threads: million ➙ billion Clock Rate of Core No significant increase Memory per Core 1-4 GB ➙ 10s–100s of MB Aggressive Fault Management in HW and SW Power Consumption 10 MW ➙ 40 MW – 150 MW Petascale Summer School 6-9 July 2010 Urbana, Illinois

24. Take Home Lessons Examine New Computing Technologies Computers of future will be based on many-core chips Details TBD, but may be heterogeneous Focus on Scalable Algorithms Only significant speed gains in future will come through increased parallelization Explore New Programming Models Computing systems will be (are!) collections of SMPs Need to assess and improve MPI/OpenMP, UPC, CAF Enhance Reliability Systems level (e.g., virtualization) Applications level Petascale Summer School 6-9 July 2010 Urbana, Illinois

25. Private Sector Program Annual Meeting 12-14 May 2008 Urbana, Illinois Questions?