1. A Pattern Language for Parallel Programming Beverly Sanders University of Florida

2. Overview of talk Background on pattern languages Motivation for a pattern language for parallel programming Tour through the pattern language via an example

3. Design Pattern High quality solution to frequently recurring problem in some domain Each pattern has a name, providing a vocabulary for discussing the solutions Written in prescribed format to allow the reader to quickly understand the solution and its context

4. A pattern format Name Also known as Context Forces Solution Examples and known uses Related patterns …

5. Pattern Language Provides domain specific advice to the designer Not a programming language Carefully structured collection of patterns Structure embodies a design methodology and leads user through the language so that complex designs can be developed using patterns

6. Parallel Programming Parallel hardware becoming increasingly mainstream Multiprocessor PCs Clusters Software to fully exploit the hardware rare (except specialized area of high performance computing) Can a pattern language providing guidance for the entire design processes help lower the barrier to parallel programming?

7. Target audience for our pattern language Primary Experienced programmer but not necessarily with parallel programming Has good understanding of the problem to be solved, its important data structures, etc. Also Researchers in new parallel programming models and environments May be able to use patterns as basis for evaluating qualitative aspects of new programming models

8. Structure of the pattern language 4 Design spaces Finding Concurrency Help designer expose exploitable concurrency—find high level task and data decomposition Algorithm Structure Help designer map tasks to processes or threads to best take advantage of the potential concurrency

9. Structure of the pattern language, continued Supporting Structures Code structuring patterns Distributed and thread-safe data structures Implementation Mechanisms Low level mechanisms used to write parallel programs 3 categories of mechanisms  UE (process/thread) Management  Synchronization  Communication Not really patterns, included for completeness We discuss OpenMP, MPI, and Java

10. Example: Molecular dynamics Simulate motion in large molecular system Example application: how protein interacts with drug Forces Bonded forces within a molecule Non-bonded forces between molecules Not tractable N 2 Use cutoff method—only consider forces from neighbors that are close enough

11. Sequential Molecular dynamics simulation real atoms(3,N) real force(3,N) int neighbors(2,M) loop over time steps Compute bonded forces Compute neighbors Compute long range forces Update position … end loop

12. Decomposition Patterns Finding Concurrency Design Space Dependency Analysis Patterns Design Evaluation

13. Decomposition Patterns Finding Concurrency Design Space Dependency Analysis Patterns Design Evaluation Task DecompositionData Decomposition

14. Molecular dynamics decomposition Each function is a loop over atoms Suggests task decomposition with each iteration corresponding to a loop iteration (or atom) tasks for nonbonded forces tasks for long range forces tasks to update positions tasks to compute neighbor list Data shared between the tasks

15. Decomposition Patterns Finding Concurrency Design Space Dependency Analysis Patterns Design Evaluation Group tasksOrder tasksData Sharing

16. Molecular dynamics dependency analysis Bonded forces Neighbor list Update position Long-range forces next time step

17. Molecular dynamics dependency analysis Bonded forces Neighbor list Update position Long-range forces next time step atoms(3,N) forces(3,N) neighbors Read Write Accumulate

18. Decomposition Patterns Finding Concurrency Design Space Dependency Analysis Patterns Design Evaluation Suitability for target platform Design Quality (flexibility, efficiency, simplicity) Preparation for the next phase

19. Design evaluation for molecular dynamics Target architecture for example: distributed memory cluster, message passing Data sharing has enough special properties (read only, accumulate, temporal constraints) that we will be able to make it work in a distributed memory environment. Design seems OK, move to next design space

20. Algorithm structure design space Map tasks to Units of Execution (thread or process) Target platform properties number of UEs communication between UEs Major organizing principle

21. Organize by tasks? Recursive? Task Parallelism Divide and Conquer yes no

22. Organize by data? Recursive? Geometric Decomposition Recursive Data yes no

23. Organize by ordering? Regular? Event-based Coordination Pipeline yes no

24. Algorithm structure for molecular dynamics Organized by task Task decomposition pattern Granularity: decide bond forces not worth parallelizing now. Load balancing: static OK, partition iterations of original loop (over atoms) to UEs Termination: easy since number of UEs can be determined in advance

25. Separable dependencies Multiple tasks update force array concurrently Certain types of dependencies can be “separated” from concurrent part of computation Each UE gets local copy of data Update local copy Reduce results (combine results using associative operator)

26. Supporting Structures A intermediate stage between algorithm structures and implementation mechanisms (Similar level to GOF) Program structuring patterns SPMD, Fork/Join, Loop Parallelism, Master/Worker Data structures Shared queue, Distributed array, Shared data

27. SPMD Shared program multiple data. Each EU executes exactly the same program Uses process ID to determine behavior Issues: replicate or partition data, computation?

28. Replicate or partition data in MD? Replicate atoms, force. Partition neighbor list Duplicate non-parallelized parts of computation, or designate one process? Duplicate computation, except I/O.

29. Parallel Simulation real atoms(3,N) real force(3,N) int neighbors(2,M) myID = getProcessID nprocs = getNumberProcesses loop over time steps Compute bonded forces //replicate computation Compute neighbors //only for atoms assigned to myID Compute long range forces globalSum(N,&forces) //reduction to combine all force arrays Update position … end loop if (myid == printID) printResults

30. Implementation Mechanisms Describes low level mechanisms used to write parallel programs 3 categories of mechanisms UE (process/thread) Management Synchronization Communication Not in pattern format We discuss OpenMP, MPI, and Java

31. Implement Simulation For our target platform, would probably choose MPI Need standard code for initialization, MPI specific reduction operator

32. Evolution Project started in 1998 Current version described in book Patterns for Parallel Programming A pattern language should not be considered a static document: Evaluate and revise Extend with new patterns: new parallel programming models, specific application domains