1. COMP7330/7336 Advanced Parallel and Distributed Computing Midterm Exam - Review Dr. Xiao Qin Auburn University http://www.eng.auburn.edu/~xqin xqin@auburn.edu Slides are adopted from Drs. Ananth Grama, Anshul Gupta, George Karypis, and Vipin Kumar 2

2. MPI: the Message Passing Interface MPI defines a standard library for message-passing that can be used to develop portable message- passing programs using either C or Fortran. The MPI standard defines both the syntax as well as the semantics of a core set of library routines. Vendor implementations of MPI are available on almost all commercial parallel computers. It is possible to write fully-functional message- passing programs by using only the six routines. 3

3. Starting and Terminating the MPI Library MPI_Init is called prior to any calls to other MPI routines. Its purpose is to initialize the MPI environment. MPI_Finalize is called at the end of the computation, and it performs various clean-up tasks to terminate the MPI environment. The prototypes of these two functions are: int MPI_Init(int *argc, char ***argv) int MPI_Finalize() MPI_Init also strips off any MPI related command-line arguments. All MPI routines, data-types, and constants are prefixed by “ MPI _”. The return code for successful completion is MPI_SUCCESS. 4

4. Communicators A communicator defines a communication domain - a set of processes that are allowed to communicate with each other. Information about communication domains is stored in variables of type MPI_Comm. Communicators are used as arguments to all message transfer MPI routines. A process can belong to many different (possibly overlapping) communication domains. MPI defines a default communicator called MPI_COMM_WORLD which includes all the processes. 5

5. Setting Up Your MPI Development Environment 6

6. Sending and Receiving Messages The basic functions for sending and receiving messages in MPI are the MPI_Send and MPI_Recv, respectively. The calling sequences of these routines are as follows: int MPI_Send(void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm) int MPI_Recv(void *buf, int count, MPI_Datatype datatype, int source, int tag, MPI_Comm comm, MPI_Status *status) MPI provides equivalent datatypes for all C datatypes. This is done for portability reasons. The datatype MPI_BYTE corresponds to a byte (8 bits) and MPI_PACKED corresponds to a collection of data items that has been created by packing non-contiguous data. The message-tag can take values ranging from zero up to the MPI defined constant MPI_TAG_UB. 7

7. Collective Communication Operations The barrier synchronization operation is performed in MPI using: int MPI_Barrier(MPI_Comm comm) The one-to-all broadcast operation is: int MPI_Bcast(void *buf, int count, MPI_Datatype datatype, int source, MPI_Comm comm) The all-to-one reduction operation is: int MPI_Reduce(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int target, MPI_Comm comm) 8

8. Overlapping Communication with Computation In order to overlap communication with computation, MPI provides a pair of functions for performing non-blocking send and receive operations. int MPI_Isend(void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm, MPI_Request *request) int MPI_Irecv(void *buf, int count, MPI_Datatype datatype, int source, int tag, MPI_Comm comm, MPI_Request *request) These operations return before the operations have been completed. Function MPI_Test tests whether or not the non-blocking send or receive operation identified by its request has finished. int MPI_Test(MPI_Request *request, int *flag, MPI_Status *status) MPI_Wait waits for the operation to complete. int MPI_Wait(MPI_Request *request, MPI_Status *status) 9

9. Avoiding Deadlocks Using non-blocking operations remove most deadlocks. Consider: int a[10], b[10], myrank; MPI_Status status;... MPI_Comm_rank(MPI_COMM_WORLD, &myrank); if (myrank == 0) { MPI_Send(a, 10, MPI_INT, 1, 1, MPI_COMM_WORLD); MPI_Send(b, 10, MPI_INT, 1, 2, MPI_COMM_WORLD); } else if (myrank == 1) { MPI_Recv(b, 10, MPI_INT, 0, 2, &status, MPI_COMM_WORLD); MPI_Recv(a, 10, MPI_INT, 0, 1, &status, MPI_COMM_WORLD); }... Replacing either the send or the receive operations with non-blocking counterparts fixes this deadlock. 10

10. Thread Basics All memory in the logical machine model of a thread is globally accessible to every thread. The stack corresponding to the function call is generally treated as being local to the thread for liveness reasons. A logical machine model with both global memory (default) and local memory (stacks). Such a flat model may result in very poor performance since memory is physically distributed in typical machines. 11

11. Threads have some advantages of (multi) processes Creation/Termination 12 Less time to create a new thread than a process, because the newly created thread uses the current process address space. Less time to terminate a thread than a process. (Why?)

12. Threads have some advantages of (multi) processes switch and communication 13 Less time to switch between two threads within the same process, partly because the newly created thread uses the current process address space. Less communication overheads -- communicating between the threads of one process is simple because the threads share everything: address space, in particular. So, data produced by one thread is immediately available to all the other threads.

13. The POSIX Thread API Commonly referred to as Pthreads, POSIX has emerged as the standard threads API, supported by most vendors. The concepts discussed here are largely independent of the API and can be used for programming with other thread APIs (NT threads, Solaris threads, Java threads, etc.) as well. 14

14. Thread Basics: Creation Pthreads provides two basic functions for specifying concurrency in a program: #include int pthread_create ( pthread_t *thread_handle, const pthread_attr_t *attribute, void * (*thread_function)(void *), void *arg); int pthread_join ( pthread_t thread, void **ptr); The function pthread_create invokes function thread_function as a thread 15

15. Thread Basics: Termination A thread can terminate in three ways : If the thread returns from its start routine. pthread_join() If it is canceled by some other thread. The function used here is pthread_cancel(). If its calls pthread_exit() function from within itself. 16

16. Synchronization Primitives in Pthreads When multiple threads attempt to manipulate the same data item, the results can often be incoherent if proper care is not taken to synchronize them. Consider: /* each thread tries to update variable best_cost as follows */ if (my_cost < best_cost) best_cost = my_cost; Assume that there are two threads, the initial value of best_cost is 100, and the values of my_cost are 50 and 75 at threads t1 and t2. Depending on the schedule of the threads, the value of best_cost could be 50 or 75! The value 75 does not correspond to any serialization of the threads. 17

17. Mutual Exclusion The Pthreads API provides the following functions for handling mutex-locks: int pthread_mutex_lock ( pthread_mutex_t *mutex_lock); int pthread_mutex_unlock ( pthread_mutex_t *mutex_lock); int pthread_mutex_init ( pthread_mutex_t *mutex_lock, const pthread_mutexattr_t *lock_attr); 18

18. Producer-Consumer Using Locks The producer-consumer scenario imposes the following constraints: The producer thread must not overwrite the shared buffer when the previous task has not been picked up by a consumer thread. The consumer threads must not pick up tasks until there is something present in the shared data structure. Individual consumer threads should pick up tasks one at a time. 19

19. Types of Mutexes Three types: – normal, – recursive, and – error-check. Normal mutex deadlocks if a thread that already has a lock tries a second lock on it. A recursive mutex allows a single thread to lock a mutex as many times as it wants. It simply increments a count on the number of locks. A lock is relinquished by a thread when the count becomes zero. An error check mutex reports an error when a thread with a lock tries to lock it again (as opposed to deadlocking in the first case, or granting the lock, as in the second case). The type of the mutex can be set in the attributes object before it is passed at time of initialization. 20

20. Overheads of Locking Locks represent serialization points. What happens if we encapsulate large segments of the program within locks? Reduce idling overhead: int pthread_mutex_trylock ( pthread_mutex_t *mutex_lock); pthread_mutex_trylock is typically much faster than pthread_mutex_lock on typical systems since it does not have to deal with queues associated with locks for multiple threads waiting on the lock. 21

21. Condition Variables for Synchronization Allows a thread to block itself until specified data reaches a predefined state. A condition variable is associated with this predicate. When the predicate becomes true, the condition variable is used to signal one or more threads waiting on the condition. A single condition variable may be associated with more than one predicate. Has a mutex associated with it. (Why?) A thread locks this mutex and tests the predicate defined on the shared variable. If the predicate is not true, the thread waits on the condition variable associated with the predicate using the function pthread_cond_wait. 22

22. Condition Variables for Synchronization Pthreads provides the following functions for condition variables: int pthread_cond_init(pthread_cond_t *cond, const pthread_condattr_t *attr); int pthread_cond_wait(pthread_cond_t *cond, pthread_mutex_t *mutex); int pthread_cond_signal(pthread_cond_t *cond); int pthread_cond_broadcast(pthread_cond_t *cond); int pthread_cond_destroy(pthread_cond_t *cond); 23

23. Producer-Consumer Using Condition Variables void *producer(void *producer_thread_data) { int inserted; while (!done()) { create_task(); pthread_mutex_lock(&task_queue_cond_lock); while (task_available == 1) pthread_cond_wait(&cond_queue_empty, &task_queue_cond_lock); insert_into_queue(); task_available = 1; pthread_cond_signal(&cond_queue_full); pthread_mutex_unlock(&task_queue_cond_lock); } 24

24. What are the differences between Condition Variable (CV) and Mutex? CV is high-level synchronization primitive CV = lock + signaling mechanism CV: like a stop light (for collaboration) Mutex: like a gate or yield sign (for competition) 25 Reference: http://stackoverflow.com/questions/1055398/differences-between-conditional-variables-mutexes- and-locks http://stackoverflow.com/questions/1055398/differences-between-conditional-variables-mutexes- and-locks

25. Read-Write Locks In many applications, a data structure is read frequently but written infrequently. For such applications, we should use read-write locks. A read lock is granted when there are other threads that may already have read locks. If there is a write lock on the data (or if there are queued write locks), the thread performs a condition wait. If there are multiple threads requesting a write lock, they must perform a condition wait. With this description, we can design functions for (What lock/unlock functions?) -read locks mylib_rwlock_rlock, -write locks mylib_rwlock_wlock, and -unlocking mylib_rwlock_unlock. 26

26. Read-Write Locks Can you implement mylib_rwlock_init ? typedef struct { int readers; int writer; pthread_cond_t readers_proceed; pthread_cond_t writer_proceed; int pending_writers; pthread_mutex_t read_write_lock; } mylib_rwlock_t; void mylib_rwlock_init (mylib_rwlock_t *l) { l -> readers = l -> writer = l -> pending_writers = 0; pthread_mutex_init(&(l -> read_write_lock), NULL); pthread_cond_init(&(l -> readers_proceed), NULL); pthread_cond_init(&(l -> writer_proceed), NULL); } 27

27. Read-Write Locks Can you implement mylib_rwlock_rlock ? void mylib_rwlock_rlock(mylib_rwlock_t *l) { /* if there is a write lock or pending writers, perform condition wait.. else increment count of readers and grant read lock */ pthread_mutex_lock(&(l -> read_write_lock)); while ((l -> pending_writers > 0) || (l -> writer > 0)) pthread_cond_wait(&(l -> readers_proceed), &(l -> read_write_lock)); l -> readers ++; pthread_mutex_unlock(&(l -> read_write_lock)); } 28

28. Read-Write Locks Can you implement mylib_rwlock_wlock ? void mylib_rwlock_wlock(mylib_rwlock_t *l) { /* if there are readers or writers, increment pending writers count and wait. On being woken, decrement pending writers count and increment writer count */ pthread_mutex_lock(&(l -> read_write_lock)); while ((l -> writer > 0) || (l -> readers > 0)) { l -> pending_writers ++; pthread_cond_wait(&(l -> writer_proceed), &(l -> read_write_lock)); } l -> pending_writers --; l -> writer ++; pthread_mutex_unlock(&(l -> read_write_lock)); } 29

29. Read-Write Locks Can you implement mylib_rwlock_unlock ? void mylib_rwlock_unlock(mylib_rwlock_t *l) { /* if there is a write lock then unlock, else if there are read locks, decrement count of read locks. If the count is 0 and there is a pending writer, let it through, else if there are pending readers, let them all go through */ pthread_mutex_lock(&(l -> read_write_lock)); if (l -> writer > 0) l -> writer = 0; else if (l -> readers > 0) l -> readers --; pthread_mutex_unlock(&(l -> read_write_lock)); if ((l -> readers == 0) && (l -> pending_writers > 0)) pthread_cond_signal(&(l -> writer_proceed)); else if (l -> readers > 0) pthread_cond_broadcast(&(l -> readers_proceed)); } 30

30. Barriers A barrier holds a thread until all threads participating in the barrier have reached it. 31

31. Barriers – Data Structure A single integer is used to keep track of the number of threads that have reached the barrier. If the count is less than the total number of threads, the threads execute a condition wait. The last thread entering (and setting the count to the number of threads) wakes up all the threads using a condition broadcast. 32

32. Data Structure of Barriers typedef struct { pthread_mutex_t count_lock; pthread_cond_t ok_to_proceed; int count; } mylib_barrier_t; Barriers can be implemented using a counter, a mutex and a condition variable. A single integer is used to keep track of the number of threads that have reached the barrier. 33

33. How to Improve Performance of Barriers? This implementation of a barrier can be speeded up using multiple barrier variables organized in a tree. We use n/2 condition variable-mutex pairs for implementing a barrier for n threads. At the lowest level, threads are paired up and each pair of threads shares a single condition variable- mutex pair. Once both threads arrive, one of the two moves on, the other one waits. This process repeats up the tree. This is also called a log barrier and its runtime grows as O(log p). 34

34. OpenMP: a Standard for Directive Based Parallel Programming A directive-based API that can be used with FORTRAN, C, and C++ for programming shared address space machines. Provide support for concurrency, synchronization, and data handling while obviating the need for explicitly setting up mutexes, condition variables, data scope, and initialization. 35

35. OpenMP Programming Model A sample OpenMP program along with its Pthreads translation that might be performed by an OpenMP compiler. 36

36. Reduction Clause in OpenMP The reduction clause specifies how multiple local copies of a variable at different threads are combined into a single copy at the master when threads exit. The usage of the reduction clause is reduction (operator: variable list). The variables in the list are implicitly specified as being private to threads. The operator can be one of +, *, -, &, |, ^, &&, and ||. #pragma omp parallel reduction(+: sum) num_threads(8) { /* compute local sums here */ } /*sum here contains sum of all local instances of sums */ 37

37. Specifying Concurrent Tasks in OpenMP The parallel directive can be used in conjunction with other directives to specify concurrency across iterations and tasks. OpenMP provides two directives - for and sections - to specify concurrent iterations and tasks. The for directive is used to split parallel iteration spaces across threads. The general form of a for directive is as follows: #pragma omp parallel for [clause list] /* for loop */ The clauses that can be used in this context are: private, firstprivate, lastprivate, reduction, schedule, nowait, and ordered. 38

38. The sections Directive: Example #pragma omp parallel { #pragma omp sections { #pragma omp section { taskA(); } #pragma omp section { taskB(); } #pragma omp section { taskC(); } 39

39. Synchronization Constructs in OpenMP demo: barrier.c The barrier directive: causes threads encountering the barrier to wait until all the other threads in the same team have encountered the barrier. #pragma omp barrier 40

40. Map-Reduce Framework

41. MapReduce – Two Phases Programmers specify two functions: map (k, v) → * reduce (k’, v’) → * – All values with the same key are reduced together The execution framework handles everything else… Not quite…usually, programmers also specify: combine (k’, v’) → * – Mini-reducers that run in memory after the map phase – Used as an optimization to reduce network traffic partition (k’, number of partitions) → partition for k’ – Often a simple hash of the key, e.g., hash(k’) mod n – Divides up key space for parallel reduce operations 42

42. Word Count: Illustrated map(key=url, val=contents): For each word w in contents, emit (w, “1”) reduce(key=word, values=uniq_counts): Sum all “1”s in values list Emit result “(word, sum)” see bob throw see spot run see1 bob1 run1 see 1 spot 1 throw1 bob1 run1 see 2 spot 1 throw1 43