1. 2/4/200408/29/2002CS267 Lecure 51 CS 267: Introduction to Parallel Machines and Programming Models Katherine Yelick yelick@cs.berkeley.edu http://www.cs.berkeley.edu/~yelick/cs267

2. 02/04/2004CS267 Lecure 52 Outline Overview of parallel machines and programming models Shared memory Shared address space Message passing Data parallel Clusters of SMPs Trends in real machines

3. 02/04/2004CS267 Lecure 53 A generic parallel architecture P P PP Interconnection Network MMMM ° Where is the memory physically located? Memory

4. 02/04/2004CS267 Lecure 54 Parallel Programming Models Control How is parallelism created? What orderings exist between operations? How do different threads of control synchronize? Data What data is private vs. shared? How is logically shared data accessed or communicated? Operations What are the atomic (indivisible) operations? Cost How do we account for the cost of each of the above?

5. 02/04/2004CS267 Lecure 55 Simple Example Consider a sum of an array function: Parallel Decomposition: Each evaluation and each partial sum is a task. Assign n/p numbers to each of p procs Each computes independent “private” results and partial sum. One (or all) collects the p partial sums and computes the global sum. Two Classes of Data: Logically Shared The original n numbers, the global sum. Logically Private The individual function evaluations. What about the individual partial sums?

6. 02/04/2004CS267 Lecure 56 Programming Model 1: Shared Memory Program is a collection of threads of control. Can be created dynamically, mid-execution, in some languages Each thread has a set of private variables, e.g., local stack variables Also a set of shared variables, e.g., static variables, shared common blocks, or global heap. Threads communicate implicitly by writing and reading shared variables. Threads coordinate by synchronizing on shared variables PnP1 P0 s s =... y =..s... Shared memory i: 2i: 5 Private memory i: 8

7. 02/04/2004CS267 Lecure 57 Shared Memory Code for Computing a Sum Thread 1 for i = 0, n/2-1 s = s + f(A[i]) Thread 2 for i = n/2, n-1 s = s + f(A[i]) static int s = 0; Problem is a race condition on variable s in the program A race condition or data race occurs when: -two processors (or two threads) access the same variable, and at least one does a write. -The accesses are concurrent (not synchronized) so they could happen simultaneously

8. 02/04/2004CS267 Lecure 58 Shared Memory Code for Computing a Sum Thread 1 …. compute f([A[i]) and put in reg0 reg1 = s reg1 = reg1 + reg0 s = reg1 … Thread 2 … compute f([A[i]) and put in reg0 reg1 = s reg1 = reg1 + reg0 s = reg1 … static int s = 0; Assume s=27, f(A[i])=7 on Thread1 and =9 on Thread2 For this program to work, s should be 43 at the end but it may be 43, 34, or 36 The atomic operations are reads and writes Never see ½ of one number All computations happen in (private) registers 79 27 3436 34

9. 02/04/2004CS267 Lecure 59 Improved Code for Computing a Sum Thread 1 local_s1= 0 for i = 0, n/2-1 local_s1 = local_s1 + f(A[i]) s = s + local_s1 Thread 2 local_s2 = 0 for i = n/2, n-1 local_s2= local_s2 + f(A[i]) s = s +local_s2 static int s = 0; Since addition is associative, it’s OK to rearrange order Most computation is on private variables -Sharing frequency is also reduced, which might improve speed -But there is still a race condition on the update of shared s -The race condition can be fixed by adding locks (only one thread can hold a lock at a time; others wait for it) static lock lk; lock(lk); unlock(lk); lock(lk); unlock(lk);

10. 02/04/2004CS267 Lecure 510 Machine Model 1a: Shared Memory P1 network/bus $ memory Processors all connected to a large shared memory. Typically called Symmetric Multiprocessors (SMPs) Sun, HP, Intel, IBM SMPs (nodes of Millennium, SP) “Local” memory is not (usually) part of the hardware abstraction. Difficulty scaling to large numbers of processors <32 processors typical Advantage: uniform memory access (UMA) Cost: much cheaper to access data in cache than main memory. P2 $ Pn $

11. 02/04/2004CS267 Lecure 511 Problems Scaling Shared Memory Why not put more processors on (with larger memory?) The memory bus becomes a bottleneck Example from a Parallel Spectral Transform Shallow Water Model (PSTSWM) demonstrates the problem Experimental results (and slide) from Pat Worley at ORNL This is an important kernel in atmospheric models 99% of the floating point operations are multiplies or adds, which generally run well on all processors But it does sweeps through memory with little reuse of operands, which exercises the memory system These experiments show serial performance, with one “copy” of the code running independently on varying numbers of procs The best case for shared memory: no sharing But the data doesn’t all fit in the registers/cache

12. 02/04/2004CS267 Lecure 512 From Pat Worley, ORNL Example: Problem in Scaling Shared Memory Performance degradation is a “smooth” function of the number of processes. No shared data between them, so there should be perfect parallelism. (Code was run for a 18 vertical levels with a range of horizontal sizes.)

13. 02/04/2004CS267 Lecure 513 Machine Model 1b: Distributed Shared Memory Memory is logically shared, but physically distributed Any processor can access any address in memory Cache lines (or pages) are passed around machine SGI Origin is canonical example (+ research machines) Scales to 100s Limitation is cache coherent protocols – need to keep cached copies of the same address consistent P1 network $ memory P2 $ Pn $ memory

14. 02/04/2004CS267 Lecure 514 Programming Model 2: Message Passing Program consists of a collection of named processes. Usually fixed at program startup time Thread of control plus local address space -- NO shared data. Logically shared data is partitioned over local processes. Processes communicate by explicit send/receive pairs Coordination is implicit in every communication event. MPI is the most common example PnP1 P0 y =..s... s: 12 i: 2 Private memory s: 14 i: 3 s: 11 i: 1 send P1,s Network receive Pn,s

15. 02/04/2004CS267 Lecure 515 Computing s = A[1]+A[2] on each processor ° First possible solution – what could go wrong? Processor 1 xlocal = A[1] send xlocal, proc2 receive xremote, proc2 s = xlocal + xremote Processor 2 xloadl = A[2] receive xremote, proc1 send xlocal, proc1 s = xlocal + xremote ° Second possible solution Processor 1 xlocal = A[1] send xlocal, proc2 receive xremote, proc2 s = xlocal + xremote Processor 2 xlocal = A[2] send xlocal, proc1 receive xremote, proc1 s = xlocal + xremote ° If send/receive acts like the telephone system? The post office?

16. 02/04/2004CS267 Lecure 516 In 2002 MPI has become the de facto standard for parallel computing The software challenge: overcoming the MPI barrier MPI created finally a standard for applications development in the HPC community Standards are always a barrier to further development The MPI standard is a least common denominator building on mid-80s technology Programming Model reflects hardware! “I am not sure how I will program a Petaflops computer, but I am sure that I will need MPI somewhere” – HDS 2001 MPI – the de facto standard

17. 02/04/2004CS267 Lecure 517 Machine Model 2a: Distributed Memory Cray T3E, NOW, IBM SP2 IBM SP-3, Millennium, CITRIS are distributed memory machines, but the nodes are SMPs. Each processor has its own memory and cache but cannot directly access another processor’s memory. Each “node” has a network interface (NI) for all communication and synchronization. interconnect P0 memory NI... P1 memory NI Pn memory NI

18. 02/04/2004CS267 Lecure 518 PC Clusters: Contributions of Beowulf An experiment in parallel computing systems Established vision of low cost, high end computing Demonstrated effectiveness of PC clusters for some (not all) classes of applications Provided networking software Conveyed findings to broad community (great PR) Tutorials and book Design standard to rally community! Standards beget: books, trained people, software … virtuous cycle Adapted from Gordon Bell, presentation at Salishan 2000

19. 02/04/2004CS267 Lecure 519 Open Source Software Model for HPC Linus's law, named after Linus Torvalds, the creator of Linux, states that "given enough eyeballs, all bugs are shallow". All source code is “open” Everyone is a tester Everything proceeds a lot faster when everyone works on one code (HPC: nothing gets done if resources are scattered) Software is or should be free (Stallman) Anyone can support and market the code for any price Zero cost software attracts users! Prevents community from losing HPC software (CM5, T3E)

20. 02/04/2004CS267 Lecure 520 Tflop/s Clusters The following are examples of clusters configured out of separate networks and processor components Shell: largest engineering/scientific cluster NCSA: 1024 processor cluster (IA64) Univ. Heidelberg cluster PNNL: announced 8 Tflops (peak) IA64 cluster from HP with Quadrics interconnect DTF in US: announced 4 clusters for a total of 13 Teraflops (peak) … But make no mistake: Itanium and McKinley are not a commodity product

21. 02/04/2004CS267 Lecure 521 Internet Computing- SETI@home Running on 500,000 PCs, ~1000 CPU Years per Day 485,821 CPU Years so far Sophisticated Data & Signal Processing Analysis Distributes Datasets from Arecibo Radio Telescope Next Step- Allen Telescope Array

22. 02/04/2004CS267 Lecure 522 Programming Model 2b: Global Addr Space Program consists of a collection of named threads. Usually fixed at program startup time Local and shared data, as in shared memory model But, shared data is partitioned over local processes Cost models says remote data is expensive Examples: UPC, Titanium, Co-Array Fortran Global Address Space programming is an intermediate point between message passing and shared memory PnP1 P0 s[myThread] =... y =..s[i]... i: 2i: 5 Private memory Shared memory i: 8 s[0]: 27s[1]: 27 s[n]: 27

23. 02/04/2004CS267 Lecure 523 Machine Model 2b: Global Address Space Cray T3D, T3E, X1, and HP Alphaserver cluster Clusters built with Quadrics, Myrinet, or Infiniband The network interface supports RDMA (Remote Direct Memory Access) NI can directly access memory without interrupting the CPU One processor can read/write memory with one-sided operations (put/get) Not just a load/store as on a shared memory machine Remote data is typically not cached locally interconnect P0 memory NI... P1 memory NI Pn memory NI Global address space may be supported in varying degrees

24. 02/04/2004CS267 Lecure 524 Programming Model 3: Data Parallel Single thread of control consisting of parallel operations. Parallel operations applied to all (or a defined subset) of a data structure, usually an array Communication is implicit in parallel operators Elegant and easy to understand and reason about Coordination is implicit – statements executed synchronously Similar to Matlab language for array operations Drawbacks: Not all problems fit this model Difficult to map onto coarse-grained machines A: fA: f sum A = array of all data fA = f(A) s = sum(fA) s:

25. 02/04/2004CS267 Lecure 525 Machine Model 3a: SIMD System A large number of (usually) small processors. A single “control processor” issues each instruction. Each processor executes the same instruction. Some processors may be turned off on some instructions. Machines are very specialized to scientific computing, so they are not popular with vendors (CM2, Maspar) Programming model can be implemented in the compiler mapping n-fold parallelism to p processors, n >> p, but it’s hard (e.g., HPF) interconnect P1 memory NI... control processor P1 memory NIP1 memory NIP1 memory NIP1 memory NI

26. 02/04/2004CS267 Lecure 526 Model 3b: Vector Machines Vector architectures are based on a single processor Multiple functional units All performing the same operation Instructions may specific large amounts of parallelism (e.g., 64-way) but hardware executes only a subset in parallel Historically important Overtaken by MPPs in the 90s Re-emerging in recent years At a large scale in the Earth Simulator (NEC SX6) and Cray X1 At a small sale in SIMD media extensions to microprocessors SSE, SSE2 (Intel: Pentium/IA64) Altivec (IBM/Motorola/Apple: PowerPC) VIS (Sun: Sparc) Key idea: Compiler does some of the difficult work of finding parallelism, so the hardware doesn’t have to

27. 02/04/2004CS267 Lecure 527 Vector Processors Vector instructions operate on a vector of elements These are specified as operations on vector registers A supercomputer vector register holds ~32-64 elts The number of elements is larger than the amount of parallel hardware, called vector pipes or lanes, say 2-4 The hardware performs a full vector operation in #elements-per-vector-register / #pipes r1r2 r3 + + … vr2 … vr1 … vr3 (logically, performs # elts adds in parallel) … vr2 … vr1 (actually, performs # pipes adds in parallel) ++++++

28. 02/04/2004CS267 Lecure 528 12.8 Gflops (64 bit) S VV S VV S VV S VV 0.5 MB $ 0.5 MB $ 0.5 MB $ 0.5 MB $ 25.6 Gflops (32 bit) To local memory and network: 2 MB Ecache At frequency of 400/800 MHz 51 GB/s 25-41 GB/s 25.6 GB/s 12.8 - 20.5 GB/s custom blocks Cray X1 Node Figure source J. Levesque, Cray Cray X1 builds a larger “virtual vector”, called an MSP 4 SSPs (each a 2-pipe vector processor) make up an MSP Compiler will (try to) vectorize/parallelize across the MSP

29. 02/04/2004CS267 Lecure 529 Cray X1: Parallel Vector Architecture Cray combines several technologies in the X1 12.8 Gflop/s Vector processors (MSP) Shared caches (unusual on earlier vector machines) 4 processor nodes sharing up to 64 GB of memory Single System Image to 4096 Processors Remote put/get between nodes (faster than MPI)

30. 02/04/2004CS267 Lecure 530 Earth Simulator Architecture Parallel Vector Architecture High speed (vector) processors High memory bandwidth (vector architecture) Fast network (new crossbar switch) Rearranging commodity parts can’t match this performance

31. 02/04/2004CS267 Lecure 531 Machine Model 4: Clusters of SMPs SMPs are the fastest commodity machine, so use them as a building block for a larger machine with a network Common names: CLUMP = Cluster of SMPs Hierarchical machines, constellations Most modern machines look like this: Millennium, IBM SPs, (not the t3e)... What is an appropriate programming model #4 ??? Treat machine as “flat”, always use message passing, even within SMP (simple, but ignores an important part of memory hierarchy). Shared memory within one SMP, but message passing outside of an SMP.

32. 02/04/2004CS267 Lecure 532 Cluster of SMP Approach A supercomputer is a stretched high-end server Parallel system is built by assembling nodes that are modest size, commercial, SMP servers – just put more of them together Image from LLNL

33. 02/04/2004CS267 Lecure 533 Outline Overview of parallel machines and programming models Shared memory Shared address space Message passing Data parallel Clusters of SMPs Trends in real machines

34. 02/04/2004CS267 Lecure 534 - Listing of the 500 most powerful Computers in the World - Yardstick: R max from Linpack Ax=b, dense problem - Updated twice a year: ISC‘xy in Germany, June xy SC‘xy in USA, November xy - All data available from www.top500.org Size Rate TPP performance TOP500

35. 02/04/2004CS267 Lecure 535 TOP500 list - Data shown ManufacturerManufacturer or vendor Computer Type indicated by manufacturer or vendor Installation SiteCustomer LocationLocation and country YearYear of installation/last major update Customer SegmentAcademic,Research,Industry,Vendor,Class. # ProcessorsNumber of processors R max Maxmimal LINPACK performance achieved R peak Theoretical peak performance N max Problemsize for achieving R max N 1/2 Problemsize for achieving half of R max N world Position within the TOP500 ranking

36. 02/04/2004CS267 Lecure 536 22nd List: The TOP10

37. 02/04/2004CS267 Lecure 537 Continents Performance

38. 02/04/2004CS267 Lecure 538 Continents Performance

39. 02/04/2004CS267 Lecure 539 Customer Types

40. 02/04/2004CS267 Lecure 540 Manufacturers

41. 02/04/2004CS267 Lecure 541 Manufacturers Performance

42. 02/04/2004CS267 Lecure 542 Processor Types

43. 02/04/2004CS267 Lecure 543 Architectures

44. 02/04/2004CS267 Lecure 544 NOW – Clusters

45. 02/04/2004CS267 Lecure 545 Analysis of TOP500 Data Annual performance growth about a factor of 1.82 Two factors contribute almost equally to the annual total performance growth Processor number grows per year on the average by a factor of 1.30 and the Processor performance grows by 1.40 compared to 1.58 of Moore's Law Strohmaier, Dongarra, Meuer, and Simon, Parallel Computing 25, 1999, pp 1517-1544.

46. 02/04/2004CS267 Lecure 546 Summary Historically, each parallel machine was unique, along with its programming model and programming language. It was necessary to throw away software and start over with each new kind of machine. Now we distinguish the programming model from the underlying machine, so we can write portably correct codes that run on many machines. MPI now the most portable option, but can be tedious. Writing portably fast code requires tuning for the architecture. Algorithm design challenge is to make this process easy. Example: picking a blocksize, not rewriting whole algorithm.

47. 02/04/2004CS267 Lecure 547 Reading Assignment Extra reading for today Cray X1 http://www.sc-conference.org/sc2003/paperpdfs/pap183.pdf Clusters http://www.mirror.ac.uk/sites/www.beowulf.org/papers/ICPP95/ "Parallel Computer Architecture: A Hardware/Software Approach" by Culler, Singh, and Gupta, Chapter 1.Parallel Computer Architecture: A Hardware/Software Approach Next week: Current high performance architectures Shared memory (for Monday) Memory Consistency and Event Ordering in Scalable Shared- Memory Multiprocessors, Gharachorloo et al, Proceedings of the International symposium on Computer Architecture, 1990.Memory Consistency and Event Ordering in Scalable Shared- Memory Multiprocessors Or read about the Altix system on the web (www.sgi.com) Blue Gene L (for Wednesday) http://sc-2002.org/paperpdfs/pap.pap207.pdf