1. Parallel Computers 1 The PRAM Model for Parallel Computation (Chapter 2) References:[2, Akl, Ch 2], [3, Quinn, Ch 2], from references listed for Chapter 1, plus the following new reference: –[5] “Introduction to Algorithms” by Cormen, Leisterson, and Rivest, First (older) edition, 1990, McGraw Hill and MIT Press, Chapter 30 on parallel algorithms. PRAM (Parallel Random Access Machine) is the earliest and best-known model for parallel computing. –A natural extension of the RAM sequential model –Has more algorithms than probably all of the other models combined. The RAM Model (Random Access Machine) consists of –A memory with M locations. Size of M is as large as needed. –A processor operating under the control of a sequential program. It can load data from memory store date into memory execute arithmetic & logical computations on data. –A memory access unit (MAU) that creates a path from the processor to an arbitrary memory location. Sequential Algorithm Steps –A READ phase in which the processor reads datum from a memory location and copies it into a register. –A COMPUTE phase in which a processor performs a basic operation on data from one or two of its registers. –A WRITE phase in which the processor copies the contents of an internal register into a memory location.

2. Parallel Computers 2 PRAM Model Description –Let P 1, P 2,..., P n be identical processors –Assume these processors have a common memory with M memory locations with M  N. –Each P i has a MAU that allows it to access each of the M memory locations. –A processor P i sends data to a processor P k by storing the data in a memory location that P k can read at a later time. –The model allows each processor to have its own algorithm and to run asynchronously. –In many applications, all processors run the same algorithm synchronously. Restricted model called synchronous PRAM Algorithm steps have 3 or less phases –READ Phase: Up to n processors may read up to n memory locations simultaneously. –COMPUTE Phase: Up to n processors perform basic arithmetic/logical operations on their local data. –WRITE phase: Up to n processors write simultaneously into up to n memory locations.

3. Parallel Computers 3 –Each processor knows its own ID and algorithms can use processor IDs to control the actions of the processors. Assumed for most parallel models PRAM Memory Access Methods –Exclusive Read (ER): Two or more processors can not simultaneously read the same memory location. –Concurrent Read (CR): Any number of processors can read the same memory location simultaneously. –Exclusive Write (EW): Two or more processors can not write to the same memory location simultaneously. –Concurrent Write (CW): Any number of processors can write to the same memory location simultaneously. Variants of Concurrent Write: –Priority CW: The processor with the highest priority writes its value into a memory location. –Common CW: Processors writing to a common memory location succeed only if they write the same value. –Arbitrary CW: When more than one value is written to the same location, any one of these values (e.g., one with lowest processor ID) is stored in memory

4. Parallel Computers 4 –Random CW: One of the processors is randomly selected write its value into memory. –Combining CW: The values of all the processors trying to write to a memory location are combined into a single value and stored into the memory location. Some possible functions for combining numerical values are SUM, PRODUCT, MAXIMUM, MINIMUM. Some possible functions for combining boolean values are AND, INCLUSIVE-OR, EXCLUSIVE-OR, etc. PRAM sometimes called “Shared Memory SIMD” (Selim Akl, Design & Analysis of Parallel Algorithms, Ch. 1, Prentice Hall, 1989.) –Assumes that at each step, all active PEs execute the same instruction, each on their own datum. –An efficient MAU that allows each PE to access each memory unit is needed. Additional PRAM comments –Focuses on what communication is needed for an algorithm, but ignores means and cost of this communications. –Now considered as unbuildable & impractical due to difficulty of supporting parallel PRAM memory access requirements in constant time. –Selim Akl shows a complex but efficient MAU for all PRAM models (EREW, CRCW, etc) in that can be supported in hardware in O(lg n) time for n PEs and O(n) memory locations. (See [2. Ch.2]. Akl also shows that the sequential RAM model also requires O(lg m) hardware memory access time for m memory locations.

5. Parallel Computers 5 PRAM ALGORITHMS Primary Reference: Chapter 4 of [2, Akl] Additional References: [5, Cormen et. al., Ch 30], and Chapter 2 of [3, Quinn] Prefix computation application considered first EREW PRAM Model is assumed. A binary operation on a set S is a function  :S  S  S. Traditionally, the element  (s 1, s 2 ) is denoted as s 1  s 1. The binary operations considered for prefix computations will be assumed to be –associative: (s 1  s 2 )  s 3 = s 1  (s 2  s 3 ) Examples –Numbers: addition, multiplication, max, min. –Strings: concatentation for strings –Logical Operations: and, or, xor Note:  is not required to be commutative. Prefix Operations: Assume s 0, s 1,..., s n-1 are in S. The computation of p 0, p 1,...,p n-1 defined below is called prefix computation: p 0 = s 0 p 1 = s 0  s 1. p n-1 = s 0  s 1 ...  s n-1

6. Parallel Computers 6 Suffix computation is similar, but proceeds from right to left. A binary operation is assumed to take constant time, unless stated otherwise. The number of steps to compute p n-1 has a lower bound of  (n) since n-1 operations are required. Draw visual algorithm in Akl, Figure 4.1 (for n=8) –This algorithm is used in PRAM algorithm below. –The same algorithm used for hypercube and combinational circuit by Akl in earlier chapter. EREW PRAM Version: Assume PRAM has n processors, P 0, P 1,..., P n-1, and n is a power of 2. Initially, P i stores x i in shared memory location s i for i = 0,1,..., n-1. for j = 0 to (lg n) -1, do for i = 2 j to n-1 do h = i - 2 j s i = s h  s i endfor

7. Parallel Computers 7 Analysis: –Running time is t(n) =  (lg n) –Cost is c(n) = p(n)  t(n) =  (n lg n) –Note not cost optimal, as RAM takes  (n) Cost-Optimal EREW PRAM Prefix Algorithm –In order to make the above algorithm optimal, we must reduce the cost by a factor of lg n. –In this case, it is easier to reduce the nr of processors by a factor of lg n. –Let k =  lg n  and m =  n/k  –The input sequence X = (x 0, x 1,..., x n-1 ) is partitioned into m subsequences Y 0, Y 1,...., Y m-1 with k items in each subsequence. While Y m-1 may have fewer than k items, without loss of generality (WLOG) we may assume that it has k items here. –The subsequences then have the form, Y i = (x i*k, x i*k+1,..., x i*k+k-1 )

8. Parallel Computers 8 Algorithm PRAM Prefix Computation (X, ,S) –Step 1: Each processor P i computes the prefix sum of the sequence Y i = (x i*k, x i*k+1,..., x i*k+k-1 ) using the RAM prefix algorithm and stores x i*k + x i*k+1 + …. + x i*k+j, in s i*k+j. –Step 2: All m PEs execute the preceding PRAM prefix algorithm on the sequence (s k-1, s 2k-1,..., s n-1 ), replacing s ik-1 with s k-1 ...  s ik-1. – Step 3: Finally, all P i for 1  i  m-1 adjust their partial value sums for all but the final term in their partial sum subsequence by performing the computation s ik+j  s ik+j  s ik-1 for 0  j  k-2. Analysis: –Step 1 takes O(lg n) = O(k) time. –Step 2 takes  (lg m) =  (lg n/k) = O(lg n- lg k) =  (lg n - lg lg n) =  (lg n) =  (k) –Step 3 takes O(k) time. (lg n) and its cost is  ((lg n)  n/(lg n)) =  (n) –The overall time for this algorithm is  (n). The combined pseudocode version of this algorithm is given on pg 155 of [2].

9. Parallel Computers 9 §4.6 Array Packing Problem: Assume that we have –an array of n elements, X = {x 1, x 2,..., x n } –Some array elements are marked (or distinguished). The requirements of this problem are to –pack the marked elements in the front part of the array. –maintain the original order between the marked elements. –place the remaining elements in the back of the array. –also, maintain the original order between the unmarked elements. Sequential solution: –Uses a technique similar to quicksort. –Use two pointers q (initially 1) and r (initially n). –Pointer q advances to the right until it hits an unmarked element. –Next, r advances to the left until it hits a marked element. –The elements at position q and r are switched and the preceding algorithm is repeated.

10. Parallel Computers 10 –This process terminates when q  r. –This requires O(n) time, which is optimal. An EREW PRAM Algorithm for Array Packing 1.Set s i in P i to 1 if x i is marked and set s i = 0 otherwise. 2. Perform a prefix sum on S to obtain the destination d i = s i for each marked x i. 3. All PEs set m = s n, the nr of marked elements. 4. Reset s i = 0 if x i is marked and s i = 1 otherwise. 5. Perform a prefix sum on S and set d i = s i + m for each unmarked x i. 6. Each P i copies array element x i into address d i in X. Algorithm analysis: –Assume n/lg(n) processors are used above. –Each prefix sum requires O(lg n) time. –The EREW broadcast in Step 3 requires O(lg n) time using either 1.a binary tree in memory (See [4]). 2.or a prefix sum on sequence b 1,…,b n with b 1 = a n and b i = 0 for 1< i  n ) –All and other steps require constant time. –Runs in O(lg n) time and is cost optimal. Note: There many applications for this algorithm.

11. Parallel Computers 11 An Optimal PRAM Sort Two references are listed below. The book by JaJa may be referenced in the future and is a well-known textbook devoted to PRAM algorithm. [6] Joseph JaJa, An Introduction to Parallel Algorithms, Addison Wesley, pgs 160-173. [7] R. Cole, Parallel Merge Sort, SIAM Journal on Computing, Vol. 17, 1988, pp. 770-785. Cole’s Merge Sort (for PRAM) –Cole’s Merge Sort runs in O(lg n) and requires O(n) processors, so it is cost optimal. –The Cole sort is significantly more efficient than most (if not all) other PRAM sorts. –A complete presentation for CREW PRAM is given in [6]. JaJa states that the algorithm he presents can be modified to run on EREW, but that the details are non-trivial. Akl calls this sort PRAM SORT in [2] and gives a very high level presentation of the EREW version of this algorithm in Ch. 4. –Currently, this sort is the best-known PRAM sort is usually the one cited when a cost-optimal PRAM sort using O(n) PEs is needed.

12. Parallel Computers 12 Comments about some other sorts for PRAM –A CREW PRAM algorithm that runs in O((lg n) lg lg n) time and uses O(n) processors which is much simpler is given in JaJa’s book (pg 158-160). This algorithm is shown to be work optimal. –Also, JaJa gives an O(lg n) time randomized sort for CREW PRAM on pages 465-473. With high probability, this algorithm terminates in O(lg n) time and requires O(n lg n) operations –i.e., with high-probability, this algorithm is work-optimal. –Sorting is sometimes called the “queen of the algorithms”: A speedup in the best-known sort for a parallel model usually results in a similar speedup other algorithms that use sorting.

13. Parallel Computers 13 Implementation Issues for PRAM (Overview) Reference: Chapter two of [2, Akl] A combinational circuit consists of a number of interconnected components arranged in columns called stages. Each component is a simple processor with a constant fan-in and fan-out –Fan-in: Input lines carrying data from outside world or from a previous stage. –Fan-out: Output lines carrying data to the outside world or to the next stage. Component characteristics: –Only active after Input arrives –Computes a value to be output in O(1) time usually using only simple arithmetic or logic operations. –Component is hardwired to execute its computation. Component Circuit Characteristics –Has no program –Has no feedback –Depth: The number of stages in a circuit Gives worst case running time for problem

14. Parallel Computers 14 –Width: Maximal number of components per stage. –Size: The total number of components –Note: size ≤ depth  width Figure 1.4 in [2, page 5] shows a combinational circuit for computing a prefix operation. Figure 2.26 in [2] shows Batcher’s odd-even merging circuit –Has 8 inputs and 19 circuits. –Its depth is 6 and width is 4. –Merges two sorted list of input values of length 4 to produce one sorted list of length 8. Two-way combinational circuits: –Sometimes used as a two-way devices –Input and output switch roles data travels from left-to-right at one time and from right-to-left at a later time. –Useful particularly for communications devices. –The circuits described in following are two- way devices and will be used to support MAU’s (memory access units).

15. Parallel Computers 15 Sorting and Merging Circuit Examples Odd-Even Merging Circuit (Fig. 2.25) –Input is two sequences of data. Length of each is n/2. Each sorted in non-decreasing order. –Output is the combined values in sorted order. –Circuit has log n stages and at most n/2 processors per stage. –Then the size is O(n lg n) –Each processor is a comparator: It receives 2 inputs and outputs the smaller of these on its top line and the other on its bottom line. Odd-Even-Merge Sorting Circuit (Fig. 2.26) –Input is sequence of n values. –Output is the sorted sequence of these values. –Has O(lg n) phases, each consisting of one or more odd-even merging circuits (stacked vertically & operating in parallel). O(lg 2 n) stages and at most n/2 processors per stage. Size is O(n lg 2 n) Odd-Even Merging and Sorting circuits are due to Prof. Ken Batcher.

16. Parallel Computers 16 Overview: Optimal Sorting Circuit (See Fig 2.27) –A complete binary tree with n leaves. Note: 1+ lg n levels and 2n-1 nodes –Non-leaf nodes are circuits (of comparators). –Each non-leaf node receives a set of m numbers Splits into m/2 smaller numbers sent to upper child circuit & remaining m/2 sent to the lower child circuit. –Sorting Circuit Characteristics Overall depth is O(lg n) and width is O(n). Overall size is O(n lg n). –Sorting Circuit is asymptotically optimal: None of O(n lg n) comparators used twice.  (n lg n) comparisons are required for sorting in the worst case. –In practice, slower than the odd-even-merge sorting circuit. The O(n lg n) size hides a very large constant of size approximately 6000. Depth is around 6,000  lg n. –This sorting circuit is a very complex circuit and its details are deferred until [2, section 3.5] OPEN QUESTION: Find an optimal sorting circuit that is practical, or show one does not exist.

17. Parallel Computers 17 Memory Access Units for RAM and PRAM A MAU for PRAM is given in [2, Akl, Ch 2.] using a combinational circuit. –Implemented as a binary tree. –The PE is connected to the root of this tree and each leaf is connected to a memory location. –If there are M memory locations for the PE then The access time (i.e., depth) is  (lg M). Circuit Width is  (M) Tree has 2M-1 =  (M) switches Size is  (M). The depth (running time) and size of above MAU are shown to be best possible in [2] using a combinational circuit. A memory access units for PRAM is also given in [2] –Discuss overview of how this MAU works. –The MAU creates a path from each PE to each memory location and handles all of the following: ER, EW, CR, CW. Handles all CW versions discussed (e.g., “combining”). –Assume n PEs and M global memory locations. –We will assume that M is a constant multiple of n. Then M =  (n). –A diagram for this MAU is given in Fig 2.30 

18. Parallel Computers 18 –Implementing MAU with the odd-even merging and sorting circuits of Batcher See Figs 2.25 and 2.26 (or examples 2.8 and 2.9) of [2]. We assumed that M is a multiple of the number of processors or M is  (n). Then the sorting circuit is the larger circuit –MAU has width O(M). –MAU has depth or running time O(lg 2 M). –MAU has size O(M lg 2 M). –Next, assume that the sorting circuit used in MAU above is replaced with the optimal sorting circuit in Figure 2.27 of [2]. Since we assume n is  (M), –MAU has width O(M) –MAU has depth or running time O(lg M) –MAU has size O(M lg M) which match the previous lower bounds (up to a constant) and hence are optimal. –Both implementations of this MAU can support all of the PRAM models using only the same resources that are required to support the weakest EREW PRAM model. –The first implementation using Batcher’s sort is practical while the second is not but is optimal.

19. Parallel Computers 19 –Note that EREW could be supported by the use of a MAU consisting of a binary tree for each PE that joins it to each memory location. Not practical, since n binary trees are required and each memory location must be connected to each of the n binary trees. END OF CHAPTER 2

20. Parallel Computers 20 TO BE ADDED IN FUTURE: EREW Broadcast and a few other basic algorithms – probably from Cormen et.al. book. A Divide & Conquer or Simulation Algorithm –To be added from [2,Ch 5], [3,Ch 2], [7,Ch 30]. –Since first round of algorithms, needs to not be overly long or challenging –Possible Candidates Merging two sorted lists [2,Ch 5] or [3] Searching an unsorted list Selection algorithm

21. Parallel Computers 21 Symbol Bar -- omit on printing   s 1                 