Multi-/Many-Core Processors Ana Lucia Varbanescu analucia@cs.vu.nl

A.L.Varbanescu - PP course @ VU Why? Ultimately, we arrived at multi-cores because we search for performance. We are interested in more performance for our codes 11/8/2018 A.L.Varbanescu - PP course @ VU

In the search for performance 11/8/2018 A.L.Varbanescu - PP course @ VU

In the search for performance We have M(o)ore transistors … How do we use them? Bigger cores Hit the walls*: power, memory, parallelism (ILP) “Dig through” ? Requires new technologies “Go around”? Multi-/many-cores *David Patterson – The Future of Computer Architecture – 2006 http://www.slidefinder.net/f/future_computer_architecture_david_patterson/6912680 11/8/2018 A.L.Varbanescu - PP course @ VU

A.L.Varbanescu - PP course @ VU Multi-/many-cores In the search for performance … Build (HW) What architectures? Evaluate (HW) What metrics? How do we measure? Use (HW + SW) What workloads? Expected performance? Program (SW (+HW)) How to program? How to optimize? Benchmark How to analyze performance? 11/8/2018 A.L.Varbanescu - PP course @ VU

Build

Choices … Core type(s): Number of cores: Memory Parallelism Fat or slim ? Vectorized (SIMD) ? Homogeneous or heterogeneous? Number of cores: Few or many ? Memory Shared-memory or distributed-memory? Parallelism SIMD/MIMD, SPMD/MPMD, … Main constraint: chip area! 11/8/2018 A.L.Varbanescu - PP course @ VU

A.L.Varbanescu - PP course @ VU A taxonomy Based on “field-of-origin”: General-purpose (GPP/GPMC) Intel, AMD Graphics (GPUs) NVIDIA, ATI Embedded systems Philips/NXP, ARM Servers Sun (Oracle), IBM Gaming/Entertainment Sony/Toshiba/IBM High Performance Computing Intel, IBM, … 11/8/2018 A.L.Varbanescu - PP course @ VU

General Purpose Processors Architecture Few fat cores Homogeneous Stand-alone Memory Shared, multi-layered; Per-core cache Programming SMP machines Both symmetrical and asymmetrical threading OS Scheduler Gain performance … MPMD, coarse-level parallelism 11/8/2018 A.L.Varbanescu - PP course @ VU

A.L.Varbanescu - PP course @ VU Intel 11/8/2018 A.L.Varbanescu - PP course @ VU

A.L.Varbanescu - PP course @ VU Intel’s next gen 11/8/2018 A.L.Varbanescu - PP course @ VU

A.L.Varbanescu - PP course @ VU AMD 11/8/2018 A.L.Varbanescu - PP course @ VU

A.L.Varbanescu - PP course @ VU AMD’s next gen 11/8/2018 A.L.Varbanescu - PP course @ VU

A.L.Varbanescu - PP course @ VU Server-side GPP-like with more HW threads Lower performance-per-thread Examples Sun UltraSPARC T2, T2+ 8 cores x 8 threads each high throughput IBM POWER7 11/8/2018 A.L.Varbanescu - PP course @ VU

Graphics Processing Units Architecture Hundreds/thousands of slim cores Homogeneous Accelerator(s) Memory Very complex hierarchy Both shared and per-core Programming Off-load model (Many) Symmetrical threads Hardware scheduler Gain performance … fine-grain parallelism, SIMT 11/8/2018 A.L.Varbanescu - PP course @ VU

A.L.Varbanescu - PP course @ VU NVIDIA G80/GT200/Fermi G80 GT200 SM = streaming multiprocessor 1 SM = 8 SP (streaming processors/CUDA cores) 1TPC = 2 x SM / 3 x SM = thread processing clusters 11/8/2018 A.L.Varbanescu - PP course @ VU

A.L.Varbanescu - PP course @ VU NVIDIA GT200 11/8/2018 A.L.Varbanescu - PP course @ VU

A.L.Varbanescu - PP course @ VU NVIDIA Fermi 11/8/2018 A.L.Varbanescu - PP course @ VU

A.L.Varbanescu - PP course @ VU ATI GPUs 11/8/2018 A.L.Varbanescu - PP course @ VU

A.L.Varbanescu - PP course @ VU Cell/B.E. Architecture Heterogeneous 8 vector-processors (SPEs) + 1 trimmed PowerPC (PPE) Accelerator or stand-alone Memory Per-core only Programming Asymmetrical multi-threading User-controlled scheduling 6 levels of parallelism, all under user control Gain performance … Fine- and coarse-grain parallelism (MPMD, SPMD) SPE-specific optimizations Scheduling 11/8/2018 A.L.Varbanescu - PP course @ VU

A.L.Varbanescu - PP course @ VU Cell/B.E. 1 x PPE 64-bit PowerPC L1: 32 KB I$+32 KB D$ L2: 512 KB 8 x SPE cores: Local mem (LS): 256 KB 128 x 128 bit vector registers Main memory access: PPE: Rd/Wr SPEs: Async DMA Available: Cell blades (QS2*): 2xCell and PS3: 1xCell (6 SPEs only) 11/8/2018 A.L.Varbanescu - PP course @ VU

Intel Single-chip Cloud Computer Architecture Tile-based many-core (48 cores) A tile is a dual-core Stand-alone / cluster Memory Per-core and per-tile Shared off-chip Programming Multi-processing with message passing User-controlled mapping/scheduling Gain performance … Coarse-grain parallelism (MPMD, SPMD) Multi-application workloads (cluster-like) 11/8/2018 A.L.Varbanescu - PP course @ VU

A.L.Varbanescu - PP course @ VU Intel SCC 11/8/2018 A.L.Varbanescu - PP course @ VU

A.L.Varbanescu - PP course @ VU Summary Computation Replace with LCPC table 11/8/2018 A.L.Varbanescu - PP course @ VU

A.L.Varbanescu - PP course @ VU Summary Memory 11/8/2018 A.L.Varbanescu - PP course @ VU

A.L.Varbanescu - PP course @ VU Take home message Variety of platforms Core types & counts Memory architecture & sizes Parallelism layers & types Scheduling Open question(s): Why so many? How many platforms do we need? Any application to run on any platform? 11/8/2018 A.L.Varbanescu - PP course @ VU

Evaluate – in theory…

HW Performance metrics Clock frequency [Hz] = Absolute HW speed(s) Memories, CPUs, interconnects Operational speed [GFLOPs] Operations per cycle Bandwidth [GB/s] memory access speed(s) differs a lot between different memories on chip Power Per core/per chip Derived metrics FLOP/Byte FLOP/Watt 11/8/2018 A.L.Varbanescu - PP course @ VU

A.L.Varbanescu - PP course @ VU Peak performance Peak = # cores * # threads_per_core * # FLOPS/cycle * clock_frequency Examples: Nehalem EX: 8 * 2 * 4 * 2.26GHz = 170 GFLOPs HD 5870: (20*16) * 5 * 0.85GHz = 1360 GFLOPs GF100: (16*32) * 2 * 1.45GHz = 1484 GFLOPs 11/8/2018 A.L.Varbanescu - PP course @ VU

On-chip memory bandwidth Registers and per-core caches - specification Shared memory: Peak_Data_Rate x Data_Bus_Width = (frequency * data_rate) * data_bus_width Example(s): Nehalem DDR3: 1.333*2*64 = 21 GB/s HD 5870: 4.800 * 256 = 153.6 GB/s Fermi: 4.200 * 384 = 201.6 GB/s 11/8/2018 A.L.Varbanescu - PP course @ VU

Off-chip memory bandwidth Depends on the interconnect Intel’s technology: QPI 25.6 GB/s AMD’s technology: HT3 19.2 GB/s Accelerators: PCI-e 1.0 or 2.0 8GB/s or 16 GB/s 11/8/2018 A.L.Varbanescu - PP course @ VU

A.L.Varbanescu - PP course @ VU Summary Cores Threads/ALUs GFLOPS BW FLOPS/Byte Cell/B.E. 8 204.80 25.6 8.0000 Nehalem EE 4 57.60 25.5 2.2588 Nehalem EX 16 170.00 63 2.6984 Niagara 32 9.33 20 0.4665 Niagara 2 64 11.20 76 0.1474 AMD Barcelona 37.00 21.4 1.7290 AMD Istanbul 6 62.40 2.4375 AMD Magny-Cours 12 124.80 4.8750 IBM Power 7 264.96 68.22 3.8839 G80 128 404.80 86.4 4.6852 GT200 30 240 933.00 141.7 6.5843 GF100 512 1484.00 201.6 7.3611 ATI Radeon 4890 160 800 680.00 124.8 5.4487 HD5870 320 1600 1360.00 153.6 8.8542 11/8/2018 A.L.Varbanescu - PP course @ VU

Absolute HW performance [1] Achieved in the optimal conditions: Processing units 100% used All parallelism 100% exploited All data transfers at maximum bandwidth Basically none – it’s even hard to build the right benchmarks … How many applications like this? 11/8/2018 A.L.Varbanescu - PP course @ VU

Evaluate – in use

A.L.Varbanescu - PP course @ VU Workloads For a new application … Design parallel algorithm Implement Optimize Benchmark Any application can run on any platform … Influence on performance portability productivity Ideally, we want a good fit! 11/8/2018 A.L.Varbanescu - PP course @ VU

A.L.Varbanescu - PP course @ VU Performance goals Hardware designer: How fast is my hardware running? End-user: How fast is my application running? End-user’s manager: How efficient is my application? Developer’s manager: How much time it takes to program it? Developer: How close can I get to the peak performance? 11/8/2018 A.L.Varbanescu - PP course @ VU

SW Performance metrics Execution time (user) Speed-up vs. best available sequential application Achieved GFLOPs (developer/user’s manager) Computational efficiency Achieved GB/s (developer) Memory efficiency Productivity and portability (developer’s manager) Production costs Maintenance costs 11/8/2018 A.L.Varbanescu - PP course @ VU

For example … Hundreds of applications to reach speed-ups of up to 2 orders of magnitude!!! Incredible performance! Or is it? 11/8/2018 A.L.Varbanescu - PP course @ VU

A.L.Varbanescu - PP course @ VU Developer Searching for peak performance … Which platform to use? What is the maximum I can achieve? And how? Performance models Amdahl’s Law Arithmetic Intensity and the Roofline model 11/8/2018 A.L.Varbanescu - PP course @ VU

A.L.Varbanescu - PP course @ VU Amdahl’s Law How can we apply Amdahl’s law for MC applications ? - Discussion 11/8/2018 A.L.Varbanescu - PP course @ VU

Arithmetic intensity (AI) AI = #OP/Byte How many operations are executed per transferred byte? Determines the boundary between compute intensive and data intensive 11/8/2018 A.L.Varbanescu - PP course @ VU

Applications AI Example: AI (RGB-to-Gray conversion) = 5/4 A r i t h m e t i c I n t e n s i t y O( N ) O( log(N) ) O( 1 ) SpMV, BLAS1,2 Stencils (PDEs) Lattice Methods FFTs Dense Linear Algebra (BLAS3) Particle Methods Is the application compute intensive or memory intensive ? Example: AI (RGB-to-Gray conversion) = 5/4 Read : 3B; Write : 1B Compute: 3 MUL + 2 ADD 11/8/2018 A.L.Varbanescu - PP course @ VU

Platform AI Is the application compute intensive or memory intensive ? RGB to Gray 11/8/2018 A.L.Varbanescu - PP course @ VU

A.L.Varbanescu - PP course @ VU The Roofline model [1] Achievable_peak = min { PeakGFLOPs, AI * streamBW } Peak GFLOPs = platform peak StreamBW = streaming bandwidth AI = application arithmetic intensity Theoretical peak values to be replaced by “real” values Without various optimizations Assumptions: Bandwidth is independent on arithmetic intensity Complete overlap of either communication or computation Computation is independent of optimization Bandwidth is independent of optimization or access pattern 1D. Lazowska, J. Zahorjan, G. Graham, K. Sevcik, “Quantitative System Performance” 11/8/2018 A.L.Varbanescu - PP course @ VU

A.L.Varbanescu - PP course @ VU The Roofline model [2] 2 1/8 flop:DRAM byte ratio attainable Gflop/s 4 8 16 32 64 128 1/4 1/2 1 Black: Theoretical peak Yellow: No streaming optimizations Green: No in-core optimizations Red: “worst case” performance zone Dashed The application 11/8/2018 A.L.Varbanescu - PP course @ VU

A.L.Varbanescu - PP course @ VU Use the Roofline model To determine what to do first to gain performance? Increase arithmetic intensity Increase streaming rate Apply in-core optimizations … and these are topics for your next lecture Samuel Williams et. al: “Roofline: an insightful visual performance model for multicore architectures” 11/8/2018 A.L.Varbanescu - PP course @ VU

A.L.Varbanescu - PP course @ VU Take home message Performance evaluation depends on goals: Execution time (users) GFLOPs and GB/s (developers) Efficiency (budget holders ) Stop tweaking when: Reach performance goal Constrained by the capabilities of the (application,platform) pair – e.g., as predicted by Roofline Choose platform to fit application Parallelism layers Arithmetic intensity Streaming capabilities 11/8/2018 A.L.Varbanescu - PP course @ VU

A.L.Varbanescu - PP course @ VU Questions Ana Lucia Varbanescu analucia@cs.vu.nl 11/8/2018 A.L.Varbanescu - PP course @ VU