2. “Sorting networks and their applications”, AFIPS Proc. of 1968 Spring Joint Computer Conference, Vol. 32, pp 307-314. http://www.cs.kent.edu/~batcher Kenneth E. Batcher Professor, Kent State University

3. SJTU2013SJTU2013 Background Sorting is fundamental Low bound of any sequential sorting algorithms is O(nlogn) Can we improve the time complexity further? –Parallel algorithms –Circuit/Network Design – Parallel Computing Models

4. SJTU2013SJTU2013 ① Bitonic Sequence 双调序列 sequence of elements {a 0, a 1, …, a n-1 } where either –(1) there exists an index, i, 0  i  n-1, such that {a 0, …, a i } is monotonically increasing, and {a i+1, …, a n-1 } is monotonically decreasing, –e.g. {1, 2, 4, 7, 6, 0} Or –(2) there exists a cyclic shift of indices so that (1) is satisfied –e.g. {8, 9, 2, 1, 0, 4}  {0, 4, 8, 9, 2, 1}

5. SJTU2013SJTU2013 Value of element aiai a 0 a 1 a 2 a 3 a 4 a 5 a 6 a 7 { 3, 5, 7, 9, 8, 6, 4, 2 } Value of element aiai a 0 a 1 a 2 a 3 a 4 a 5 a 6 a 7 { 8, 6, 4, 2, 3, 5, 7, 9} ① Bitonic Sequence : Examples

6. SJTU2013SJTU2013 Value of element aiai a 0 a 1 a 2 a 3 a 4 a 5 a 6 a 7 { 3, 5, 7, 9, 11, 13, 15, 17 } ① Bitonic Sequence : Examples Value of element aiai a 0 a 1 a 2 a 3 a 4 a 5 a 6 a 7 { 5, 3, 1, 2, 4, 6, 8, 7 }

7. SJTU2013SJTU2013 Bitonic Sort: basic idea Consider a bitonic sequence S of size n where –the first half ( {a 0, a 1, …, a n/2-1 } ) is increasing, and the second half ( {a n/2, a n/2+1, …, a n-1 } ) is decreasing Value of element aiai a 0 a 1... a n/2-1 a n/2 a n/2 +1 … a n-1

8. SJTU2013SJTU2013 ② “Bitonic Split” 双调分裂 Pair-wise min-max comparison –s 1 = {min(a 0, a n/2 ), min(a 1, a n/2+1 ), …, min(a n/2-1, a n-1 )} –s 2 = {max(a 0, a n/2 ), max(a 1, a n/2+1 ), …, max(a n/2-1, a n-1 )} a n/2 a0a0 a n/2-1 a n-1 value aiai a 0 a 1... a n/2-1 a n/2 a n/2 +1 … a n-1 Compare and exchange S2S2 S1S1 value

9. SJTU2013SJTU2013 There exists –an element b in S 1 such that all elements before b is increasing and all elements after b is decreasing –an element c in S 2 such that all elements before c is decreasing and all elements after c is increasing S 1 and S 2 –Both S 1 and S 2 are bitonic sequences –Any elements in S 1 < any elements in S 2 (because b < c and b is the maximum value in S 1 and c is the minimum value in S 2 ) S2S2 S1S1 value c b

10. SJTU2013SJTU2013 pair-wise min-max comparison e.g. { 2, 4, 6, 8, 7, 5, 3, 1} { 2, 4, 6, 8 7, 5, 3, 1 } => S1={2, 4, 3, 1} S2={7, 5, 6, 8} bitonic sequence of size 8 => 2 bitonic sequence of size 4 Compare and exchange

11. SJTU2013SJTU2013 ② Bitonic Split The split is applicable to any bitonic sequence. Need not to have the 1 st half to be increasing/decreasing and the 2 nd half to be decreasing/increasing: Bitonic(n) 2 Bitonic(n/2) Bitonic Split

12. SJTU2013SJTU2013 Sorting a bitonic sequence By using bitonic split recursively, INPUT: a bitonic sequence of size n  Phase 1: 2 bitonic sequence of size n/2  Phase 2: 4 bitonic sequence of size n/4  …  Phase (log n): n bitonic sequence of size 1  a sorted sequence can be generated by concatenating the n bitonic sequence of size 1

13. SJTU2013SJTU2013 ③ Bitonic Merge 双调合并 sort a bitonic sequence using bitonic splits 1 2 3 4 5 6 7 8 9 10111213141516 length 8 16 4 2

14. SJTU2013SJTU2013 Bitonic Merge Circuit :  BM[16]

15. SJTU2013SJTU2013 Questions ? How can we convert an unsorted sequence to a bitonic sequence ? (then, by using bitonic split recursively, a sorted sequence can be formed).

16. SJTU2013SJTU2013 1 2 3 4 5 6 7 8 9 10111213141516 length 8 4 16 Turn an unsorted sequence into a bitonic sequence: ③ Bitonic Merge (BM) Operation At every phase, sort a bitonic sequence of size 2, 4, 8, 16 into a monotonically increasing or decreased sequence

17. SJTU2013SJTU2013 Turn an unsorted sequence into a bitonic sequence

18. SJTU2013SJTU2013 1 2 3 4 5 6 7 8 9 10111213141516 length 8 4 16 ④ Bitonic Sort

19. SJTU2013SJTU2013 Sort (any ordered of) sequence Using bitonic merge repeatedly Definition: –  BM[n]: increasing bitonic merge of size n bitonic merge : sort a bitonic sequence of size n into a monotonically increasing sequence –  BM[n]: decreasing bitonic merge of size n bitonic merge that sort a bitonic sequence of size n into a monotonically decreasing sequence

20. SJTU2013SJTU2013 Steps: Divide the sequence into a group of 2 –any sequence of size 2 is a bitonic sequence: either the increasing part is of size 2 and the decreasing part is of size 0, or vice versa Using  BM[2] on a group to form an increasing sequence, and  BM[2] on the adjacent group to form an decreasing sequence Concatenate the two group to form a bitonic sequence of size 4

21. SJTU2013SJTU2013 Repeat the above steps on other groups Repeat the above steps recursively, until a bitonic sequence of size n is formed Using bitonic merge again to turn the bitonic sequence into a sorted sequence Steps:

22. SJTU2013SJTU2013 –  BM[n]: increasing bitonic merge of size n –  BM[n]: decreasing bitonic merge of size n Bitonic Sorting Circuit: BS(18)

23. SJTU2013SJTU2013 Sort (any ordered of) sequence Hence, n unsorted numbers n/2 group of 2-number bitonic sequence n/4 group of 4-number bitonic sequence … 1 group of n-number bitonic sequence a sorted sequence

24. SJTU2013SJTU2013 ⑤ Complexity of Bitonic Sort Parallel bitonic sort with n processor –The last stage of an n-element bitonic sorting need to merge n-element, and has a depth of log(n) –Other stages perform a complete sort of n/2 elements –Depth, d(n) = d(n/2) + log(n) –d(n) = 1 + 2 + 4 + … + log(n) =  (log 2 n) –Complexity: T(n) =  (log 2 n)

25. SJTU2013SJTU2013 ⑤ Complexity of Bitonic Sort Parallel sorting with a block of elements per processor –sort the local block of elements first (using any sorting algorithm such as quicksort, bitonic sort) –sort the elements among processors using parallel bitonic sort –T(n) = T(local_sort) + T(comparisons) +T(communication) Only computation time is considered here (you need to consider all communication time also)

26. SJTU2013SJTU2013 ⑥ Concluding Remarks Bitonic Sorting: Common Sense Regression to Computer Science One of 10 Most Important Papers Parallel Algorithm: Ascend/Descend –Another example: Prefix sum Network Model:

27. SJTU2013SJTU2013 27 Bitonic Sorting Network Hypercube connections! Try to Write Bitonic Sorting algorithm on hypercube.

28. SJTU2013SJTU2013 28 Bitonic Sort on Butterfly

29. SJTU2013SJTU2013 29 Bitonic Sort on Butterfly

30. SJTU2013SJTU2013 30 Bitonic Sort on Butterfly

31. SJTU2013SJTU2013 31 Bitonic Sort on Butterfly

32. SJTU2013SJTU2013 32 Bitonic Sort on Butterfly

33. SJTU2013SJTU2013 33 Bitonic Sort on Butterfly

34. SJTU2013SJTU2013 34 Bitonic Sort on Butterfly

35. SJTU2013SJTU2013 35 Bitonic Sort on Butterfly

36. SJTU2013SJTU2013 36 Bitonic Sort on Butterfly

37. SJTU2013SJTU2013 37 Bitonic Sort on Butterfly

38. SJTU2013SJTU2013 38 Bitonic Sort on Butterfly

39. SJTU2013SJTU2013 39 Bitonic Sort on Butterfly

40. SJTU2013SJTU2013 40 Bitonic Sort on Butterfly

41. SJTU2013SJTU2013 PRAM Model P1P2P3Pn … Memory Access time from any processor to any memory unit is equal It is impossible in practice So it is an ideal model for parallel computing

42. SJTU2013SJTU2013 PRAM Model Program for Sum= a(1)+a(2)+…+a(N) for i = 1 to log N for j= 1 to n/ 2 i parallel do a(j) = a(j) + a(N/ 2 i + j) endpar endfor Finally a(1) is the sum

43. SJTU2013SJTU2013 Hypercube Model Suppose node N(i) holds element a(i), where i is the value of node index x 1 x 2 …x n for i = 1 to n for j= i to n parallel do N(00…0 (x j =0) x j+1 …x n )  N(00…0 (x j =1) x j+1 …x n ); a(00…0 (x j =0) x j+1 …x n ) = a(00…0 (x j =0) x j+1 …x n ) + a(00…0 (x j =1) x j+1 …x n ) endpar endfor Finally node 00…0 holds the sum

44. SJTU2013SJTU2013 Hypercube Model Suppose node 000 holds element a(0) and 111holds element a(7) a(4) a(1) a(3) a(7) a(2) a(0) a(5) a(6) a(1)+a(5) a(3)+a(7)a(2)+a(6) a(0)+a(4) a(1)+a(5)+a(3)+a(7) +a(2)+a(6)a(0)+a(4) +a(1)+a(5)+a(3)+a(7)+a(2)+a(6)a(0)+a(4)