1. ICOM 5995: Performance Instrumentation and Visualization for High Performance Computer Systems Lecture 8 October 23, 2002 Nayda G. Santiago

2. Overview Message Passing and Shared Memory Reference Designing and Building Parallel Programs, by Ian Foster (textbook), Chapters 1, 2, and 8. Maui HPC Center site

3. Class Example Take a piece of paper. Algorithm Get your initial value. Initialize with the number of neighbors you have. Compute the average of your neighbor’s values and subtract from your value. Make this your new value. Repeat until done.

4. Class Example (cont.) Questions (Think message passing or shared memory) How do you get the values from your neighbors? Which step or iteration do they correspond to? Do you know? Do you care? How do you decide when you are done?

5. Communication for message passing Point to point Collective Patterns Broadcast Scatter Gather All Gather All to All Reduction operations

6. Point to Point Communication Most basic type of communication Send is accompanied by a matching receive Types Blocking – No processing until message transmitted Nonblocking – Continues processing, even if message is not transmitted data Memory data Memory Processor AProcessor B Network Send Data Receive Data

7. Broadcast One node has information needed by all processors. The processor sends this information to many other nodes. T1T2T3 P0 P1 P2 P3 P0 P1 P2 P3 Broadcast AAAAA

8. All of the data are initially collected on a single processor. After the scatter operation, pieces of the data are distributed on different processors. Scatter P0 P1 P2 P3 ABCD P0 P1 P2 P3 A B C D Scatter

9. The gather operation is the inverse operation to scatter: it collects pieces of the data that are distributed across a group of processors and reassembles them in the proper order on a single processor. Gather P0 P1 P2 P3 ABCD P0 P1 P2 P3 A B C D Scatter

10. Think as a gather but all processors will receive the information, not only root. All Gather P0 P1 P2 P3 ABCD P0 P1 P2 P3 A B C D All Gather ABCDABCDABCD

11. All to All Broadcast Each processor sends its unique information to all the other processors.

12. Reduction Operations Collective operation in which a single process (the root process) collects data from the other processes in a group and combines them into a single data item. Example Compute the sum of the elements of an array that is distributed over several processors. Operations other than arithmetic ones are also possible, for example, maximum and minimum, as well as various logical and bitwise operations.

13. Reduction Operations Examples