1. Computer Architecture II 1 Computer architecture II Introduction

2. Computer Architecture II 2 Recap Parallelization strategies –What to partition? –Embarrassingly Parallel Computations –Divide-and-Conquer –Pipelined Computations Application examples Parallelization steps 3 programming models –Data parallel –Shared memory –Message passing

3. Computer Architecture II 3 4 Steps in Creating a Parallel Program Decomposition of computation in tasks Assignment of tasks to processes Orchestration of data access, comm, synch. Mapping processes to processors

4. Computer Architecture II 4 Plan for today Programming for performance Amdahl’s law Partitioning for performance –Addressing decomposition and assignment Orchestration for performance

5. Computer Architecture II 5 Creating a Parallel Program Assumption: Sequential algorithm is given –Sometimes need very different algorithm, but beyond scope Pieces of the job: –Identify work that can be done in parallel –Partition work and perhaps data among processes –Manage data access, communication and synchronization –Note: work includes computation, data access and I/O Main goal: Speedup (plus low prog. effort and resource needs) Speedup (p) = For a fixed problem: Speedup (p) = Performance(p) Performance(1) Time(1) Time(p)

6. Computer Architecture II 6 Amdahl´s law Suppose a fraction f of your application is not parallelizable 1-f : parallelizable on p processors Speedup(P) = T 1 /T p <= T 1 /(f T 1 + (1-f) T 1 /p) = 1/(f + (1-f)/p) <= 1/f

7. Computer Architecture II 7 Amdahl’s Law (for 1024 processors) See: Gustafson, Montry, Benner, “Development of Parallel Methods for a 1024 Processor Hypercube”, SIAM J. Sci. Stat. Comp. 9, No. 4, 1988, pp.609.

8. Computer Architecture II 8 Amdahl´s law But: –There are many problems can be “embarrassingly” parallelized Ex: image processing, differential equation solver –In some cases the serial fraction does not increase with the problem size –Additional speedup can be achieved from additional resources (super-linear speedup due to more memory) “

9. Computer Architecture II 9 Performance Goal => Speedup Architect Goal –observe how program uses machine and improve the design to enhance performance Programmer Goal –observe how the program uses the machine and improve the implementation to enhance performance

10. Computer Architecture II 10 4 Steps in Creating a Parallel Program Decomposition of computation in tasks Assignment of tasks to processes Orchestration of data access, comm, synch. Mapping processes to processors

11. Computer Architecture II 11 Partitioning for Performance First two phases of parallelization process: decomposition & assignment Goal 1.Balancing the workload and reducing wait time at synch points 2.Reducing inherent communication 3.Reducing extra work for determining and managing a good assignment (static versus dynamic) Tensions between the 3 goals –Maximize load balance => smaller tasks => increase communication –No communication (run on 1 processor) => extreme load imbalance (all others idle) –Load balance => extra work to compute or manage the partitioning (ex. dynamic techniques)

12. Computer Architecture II 12 1. Load Balance –Work: data access, computation –Not just equal work, but must be busy at same time –Ex: Speedup ≤ 1000/400 = 2.5 Sequential Work Max Work on any Processor Speedup ≤

13. Computer Architecture II 13 1. Load balance a)Identify enough concurrency Data and functional parallelism (last class) b)Managing concurrency c)Task granularity d)Reduce communication and synchronization

14. Computer Architecture II 14 1 b) Static versus Dynamic assignment Static: before the program starts is clear who does what #pragma omp parallel for schedule(static) for(i=0;I<N;i++) { a[i] = a[i] + b[i];} Dynamic –External scheduler –Self-scheduled Each process picks a chunk of loop iterations and executes them #pragma omp parallel for schedule(dynamic,4) for(i=0;I<N;i++) { a[i] = a[i] + b[i];} Dynamic guided self-scheduling: processes take first larger chunks and then reduce this number progressively #pragma omp parallel for schedule(guided,4) for(i=0;I<N;i++) { a[i] = a[i] + b[i];}

15. Computer Architecture II 15 Dynamic Tasking with Task Queues Centralized queue: simple protocol –Problems: Communication, synchronization, contention Distributed queues: complicated protocol –Initial distribution of jobs May cause load imbalance Solution: task stealing: whom to steal from, how many tasks to steal,... –Termination detection

16. Computer Architecture II 16 1.c Task granularity Task granularity: amount of work associated with a task General rule: –Coarse-grained: often less load balance –Fine-grained: better load balance, but more overhead, often more communication and contention Processor 1 Processor 2 Processor 1 Processor 2

17. Computer Architecture II 17 1.d Reducing Serialization Synchronization for task assignment may cause serialization (for instance the access to a queue) Sequential Work Max (Work + Synch Wait Time) Speedup < Process 1 Process 2 Process 3 Synchronization pointWorkSynchronization wait time

18. Computer Architecture II 18 Reducing Serialization Event synchronization –Reduce use of conservative synchronization point-to-point instead of barriers finer granularity of access may reduce the synchronization time –But fine-grained synchronization more difficult to program, more synchronization operations. Mutual exclusion –Separate locks for separate data lock per task in task queue, not per queue finer grain => less contention/serialization, more space, less reuse –Smaller, less frequent critical sections don’t do reading/testing in critical section, only modification e.g. searching for task to dequeue outside critical section

19. Computer Architecture II 19 2. Reducing Inherent Communication Communication is expensive! Measure: communication to computation ratio Inherent communication –Determined by assignment of tasks to processes –Actual communication may be larger (artifactual) One principle: Assign tasks that access same data to same process Sequential Work Max (Work + Synch Wait Time + Comm Cost) Speedup < Synchronization pointWorkSynchronization wait time Communication Process 1 Process 2 Process 3

20. Computer Architecture II 20 Domain Decomposition Ocean Example: communicate with the neighbors, compute in the assigned domain Perimeter to Area communication-to-computation ratio (area to volume in 3-d): Depends on n,p: decreases with n, increases with p

21. Computer Architecture II 21 Domain Decomposition Communication/computation: for block, for strip –Block better –Application dependent: strip may be better in other cases 4*√p n 2*p n Best domain decomposition depends on information requirements Block versus strip decomposition:

22. Computer Architecture II 22 Finding a Domain Decomposition GOALS: load balance & low communication Static, by inspection –Must be predictable: Ocean Static, but not by inspection –Input-dependent, require analyzing input structure –E.g sparse matrix computations Semi-static (periodic repartitioning) –Characteristics change but slowly; e.g. Barnes-Hut Static or semi-static, with dynamic task stealing –Initial decomposition, but highly unpredictable; e.g ray tracing

23. Computer Architecture II 23 3. Reducing Extra Work Common sources of extra work: –Computing a good partition e.g. partitioning in Barnes-Hut –Using redundant computation to avoid communication –Task, data and process management overhead applications, languages, runtime systems, OS –Imposing structure on communication coalescing small messages Sequential Work Max (Work + Synch Wait Time + Comm Cost + Extra Work) Speedup <

24. Computer Architecture II 24 PART II: memory aware optimizations So far we have seen the parallel computer as a collection of communicating processors –Goals: balance load, reduce inherent communication and extra work –We have assumed an unlimited memory In reality the parallel computer uses a multi-cache, multi- memory system

25. Computer Architecture II 25 Memory-oriented View Multiprocessor as Extended Memory Hierarchy Levels in extended hierarchy: –Registers, caches, local memory, remote memory –Glued together by communication architecture –Levels communicate at a certain granularity of data transfer Proc Cache L2 Cache L3 Cache Memory Proc Cache L2 Cache L3 Cache Memory potential interconnects Granularity increases, access time increases Capacity increases, cost/unit decreases

26. Computer Architecture II 26 Memory-oriented view Performance depends heavily on memory hierarchy Time spent by a program (usually given in cycles) Time prog =Time compute + Time access Data access time can be reduced by: – Optimizing machine larger caches lower latency Larger bandwidth – Optimizing program temporal and spatial locality

27. Computer Architecture II 27 Artifactual Communication in Extended Hierarchy poor allocation of data across distributed memories –Data accessed by a node in the memory of another unnecessary data in a transfer unnecessary transfers due to system granularities redundant communication of data finite replication capacity (in cache or main memory)

28. Computer Architecture II 28 Replication induced artifactual communication Communication induced by finite capacity is most fundamental artifact –Like cache size and miss rate or memory traffic in uniprocessors View as three level hierarchy for simplicity –Local cache, local memory, remote memory (ignore network topology) Classify “misses” in “cache” at any level as for uniprocessors (4 “C”s) compulsory or cold misses (no size effect) capacity misses (yes) conflict or collision misses (yes) communication or coherence misses (no) – Each may be helped/hurt by large transfer granularity (spatial locality)

29. Computer Architecture II 29 Working Set Perspective –Hierarchy of working sets –At first level cache (fully assoc, one-word block), inherent to algorithm working set curve for program –Traffic from any type of miss can be local or nonlocal (communication)

30. Computer Architecture II 30 Orchestration for Performance Reducing amount of communication: –Inherent: change the partitioning (seen earlier) –Artifactual: exploit spatial, temporal locality in the memory hierarchy Techniques often similar to those on uniprocessors Structuring communication to reduce cost

31. Computer Architecture II 31 Reducing Artifactual Communication Message passing model –Communication and replication are both explicit –Even artifactual communication is in explicit messages Shared address space model –Occurs transparently due to interactions of program and system –used for explanation

32. Computer Architecture II 32 Exploiting Temporal Locality –Def: reusing of data elements already brought into cache –Structure algorithm so working sets fit into the cache often techniques to reduce inherent communication: –assign tasks accessing the same elements to the same processor schedule tasks for data reuse once assigned –Ocean Solver example: blocking Each grid element accessed 5 times First time brought into cache then reused Rewrite the loops

33. Computer Architecture II 33 Exploiting Spatial Locality Def: when a data element is accessed, its neighbors are accessed Major spatial-related causes of artifactual communication : –Conflict misses –Data distribution/layout (allocation granularity) –Fragmentation (communication granularity) –False sharing of data (coherence granularity) AVOIDING ARTIFACTUAL COMMUNICATION: keep contiguous data accessed by one processor –Fix problems by modifying data structures, or layout/alignment

34. Computer Architecture II 34 Spatial Locality Example – Repeated sweeps over 2-d grid, each time adding 1 to elements – 4-d grid to achieve spatial locality ( line processor x column processor x line index x column index)

35. Computer Architecture II 35 Tradeoffs with Inherent Communication Partitioning grid solver: blocks versus rows –Blocks have a spatial locality problem on remote data: when accessing the elements of neighboring processors whole cache blocks are fetched at column boundary –Row-wise can perform better despite worse inherent communication-to-computation ratio

36. Computer Architecture II 36 Example Performance Impact on Origin2000 OceanKernel solver