1. UMass Lowell Computer Science 91.503 Analysis of Algorithms Prof. Karen Daniels Fall, 2001 Lecture 9 Tuesday, 11/20/01 Parallel Algorithms Chapters 28, 30

2. Relevant Sections of Chapters 28-30 Ch28 Sorting Networks You’re responsible for material in this chapter that we discuss in lecture. (Note that this includes all sections 28.1 - 28.5.) Ch29 Arithmetic Circuits You’re not responsible for any of the material in this chapter. We will not be discussing it in lecture. Ch30 Algorithms for Parallel Computers You’re responsible for material in this chapter that we discuss in lecture. (Note that this includes only sections 30.1 - 30.2.)

3. Overview ä Sorting Networks ä Comparison Networks ä 0-1 Principle ä Bitonic Sorting Network ä Merging Network ä Sorting Network ä Algorithms for Parallel Computers ä PRAM Model ä Pointer Jumping ä CRCW Algorithms vs. EREW Algorithms

4. Sorting Networks Chapter 28 Comparison Networks 0-1 Principle Bitonic Sorting Network Merging Network Sorting Network

5. Comparison Networks: Definition Comparison Network only performs comparisons. Comparisons may occur in parallel. 2-input comparator: Comparison Network contains only comparators & wires. input wires output wires source: 91.503 textbook Cormen et al.

6. Comparison Networks: Definition (continued) Graph of interconnections must be acyclic. Define time using wire depth. Running Time: Comparator uses  (1) time. Input wire has depth = 0. Comparator with input wire depths d x, d y has output wire depths = Depth of comparison network = max depth of a comparator. source: 91.503 textbook Cormen et al.

7. Sorting Network: Definition ä Sorting Network: ä Comparison Network for which output sequence is monotonically increasing source: 91.503 textbook Cormen et al. Example:

8. Sorting Network: Structure Families of Comparison Networks BITONIC-SORTERs MERGERs SORTERs HALF-CLEANERs COMPARATORs BitonicSortingNetworks MergingNetworks SortingNetworks Recursive Structure “Parallel MergeSort” Strategy Sort n values in O( lg 2 n ) time

9. 0-1 Principle source: 91.503 textbook Cormen et al. If sorting network works correctly for {0,1} inputs, it works correctly on arbitrary input numbers. Proof relies on function monotonicity: allows us to limit attention to {0,1} inputs

10. 0-1 Principle (continued) Proof of Lemma 28.1: source: 91.503 textbook Cormen et al. f monotonically increasing comparator with inputs f(x), f(y) produces outputs f(min(x,y)), f(max(x,y)) Induction on wire depth

11. 0-1 Principle (continued) ä Example applying Lemma 28.1: source: 91.503 textbook Cormen et al.

12. 0-1 Principle (continued) source: 91.503 textbook Cormen et al. If sorting network works correctly for {0,1} inputs, it works correctly on arbitrary input numbers. allows us to limit attention to {0,1} inputs

13. Sorting Network: Structure Families of Comparison Networks BITONIC-SORTERs MERGERs SORTERs HALF-CLEANERs COMPARATORs BitonicSortingNetworks MergingNetworks SortingNetworks Recursive Structure “Parallel MergeSort” Strategy Sort n values in O( lg 2 n ) time

14. Bitonic Sorting Network ä Bitonic Sequence ä monotonically increases then monotonically decreases ä or can be circularly shifted to conform to this ä Example: ä Example: ä {0,1} bitonic sequence has structure:  0 i 1 j 0 k or 1 i 0 j 1 k ä Bitonic Sorter: ä comparison network that sorts bitonic {0,1} sequences ä will be used to construct Sorting Network source: 91.503 textbook Cormen et al.

15. Bitonic Sorting Network source: 91.503 textbook Cormen et al. ä Bitonic Sorter uses HALF-CLEANERs Sample inputs & outputs: HALF-CLEANER: - comparison network of depth 1 - input line i compared with line i + n/2 for i = 1,2,…,n/2 - input line i compared with line i + n/2 for i = 1,2,…,n/2

16. Bitonic Sorting Network source: 91.503 textbook Cormen et al.

17. Bitonic Sorting Network source: 91.503 textbook Cormen et al.

18. Bitonic Sorting Network source: 91.503 textbook Cormen et al. HALF- CLEANER[n] BITONIC- SORTER[n/2] Recurrence for depth of BITONIC-SORTER[n]

19. Sorting Network: Structure Families of Comparison Networks BITONIC-SORTERs MERGERs SORTERs HALF-CLEANERs COMPARATORs BitonicSortingNetworks MergingNetworks SortingNetworks Recursive Structure “Parallel MergeSort” Strategy Sort n values in O( lg 2 n ) time

20. Merging Network source: 91.503 textbook Cormen et al. Merge 2 sorted input sequences into 1 sorted output sequence. use modification of BITONIC-SORTER KEY IDEA: For sorted input sequences X, Y: XY R is bitonic can merge X, Y, using BITONIC-SORTER(XY R ) challenge: perform reversal implicitly

21. Merging Network source: 91.503 textbook Cormen et al. BITONIC- SORTER[n/2]

22. Sorting Network: Structure Families of Comparison Networks BITONIC-SORTERs MERGERs SORTERs HALF-CLEANERs COMPARATORs BitonicSortingNetworks MergingNetworks SortingNetworks Recursive Structure “Parallel MergeSort” Strategy Sort n values in O( lg 2 n ) time

23. Sorting Network source: 91.503 textbook Cormen et al. SORTER[n/2] SORTER[n/2] MERGER[n] MERGER[8] MERGER[4] MERGER[4] MERGER[2] MERGER[2] MERGER[2] MERGER[2] Recurrence for depth of SORTER[n]

24. Algorithms for Parallel Computers Chapter 30 PRAM Model Pointer Jumping CRCW Algorithms vs. EREW Algorithms

25. PRAM Model ä Need a model for parallel computing ä RAM model is serial ä Sorting network (Ch28) too restrictive ä Popular model: PRAM ä Parallel Random Access Machine source: 91.503 textbook Cormen et al.

26. PRAM Model ä Memory Access Policies: ä Common-CRCW model: ä When processors write “simultaneously” to same memory location, they write same value ä Alternatives: source: 91.503 textbook Cormen et al. Section 30.1 Section 30.2

27. Pointer Jumping: List Ranking source: 91.503 textbook Cormen et al. O( lgn ) time  ( nlgn ) work Work = time x #processors Correctness Invariant: At start of each iteration of while loop, for each object i, sum of d values for sublist headed by i = correct d[i] Running-Time Invariant: Each step of pointer jumping transforms each list into 2 interleaved lists (even, odd). List Ranking Problem: Given singly-linked list of n objects, compute, for each object, its distance from end of list:

28. Pointer Jumping: Prefix source: 91.503 textbook Cormen et al. O( lgn ) time Correctness Invariant: At end of tth iteration of while loop, kth processor stores [max(1,k-2 t +1),k] At each, if we perform prefix computation on each existing list, each object obtains correct value. start with x[i]=x k in each object i of the list Differences from LIST-RANK

29. Pointer Jumping: Euler Tour source: 91.503 textbook Cormen et al. Problem: Compute depth of each node in n-node binary tree. 1) Construct Euler Tour of a graph: (cycle traversing each edge exactly once.) O(1) time 2) Initialize values for each of processor 3 processors per node: 3) Parallel Prefix computation using + O(lgn) time

30. CRCW vs. EREW Algorithms source: 91.503 textbook Cormen et al. ä Problem where concurrent reads help: ä Find identities of tree roots in a forest

31. CRCW vs. EREW Algorithms source: 91.503 textbook Cormen et al. ä Problem where concurrent writes help: ä Find maximum element in array of real numbers

32. CRCW vs. EREW Algorithms source: 91.503 textbook Cormen et al.