Parallel Algorithms

Parallel Models u Hypercube u Butterfly u Fully Connected u Other Networks u Shared Memory v.s. Distributed Memory u SIMD v.s. MIMD

The PRAM Model u Parallel Random Access Machine u All processors act in lock-step u Number of processors is not limited u All processors have local memory u One global memory accessible to all processors u Processors must read and write global memory

A Pram Algorithm u Every Processor knows its own index (usually indicated by variable i) u Vector Sum: Read M[i] Into x; Read M[i+n] Into y; x := x + y; Write x into M[i];

Binary Fan-In Read M[i] into Largest; Write M[i] into M[i+n]; Delta := 1; For k := 1 to  lg n  Read M[i+Delta] into x; Largest := Maximum(x,Largest); Write Largest into M[i]; Delta := Delta * 2; End For

Parallel Addition Read M[i] into Total; Write 0 into M[i+n]; Delta := 1; For k := 1 to  lg n  Read M[i+Delta] into x; Total := x + Total; Write Total into M[i]; Delta := Delta * 2; End For

Pointer Jumping Read M[i] Into Total; For k := 1 to  lg n  Read Next[i] into Ptr If Ptr  0 Then Read M[Ptr] Into x; Total := Total + x; Write Total into M[i]; Read Next[Ptr] Into NewPtr Write NewPtr into Next[i] End If End For

Initialization of Next[i] If i = n Then Write 0 Into Next[i]; Else Write i+1 Into Next[i]; End If

Calculate Node Depth I 1 0 If there is a Left Child To “1” of Left Child From “-1” of Left Child

Calculate Node Depth If there is no left child

Calculate Node Depth If there is a Right Child To “1” of Right Child From “-1” of Right Child

Calculate Node Depth If there is no right child

Concurrent Reads & Writes u EREW - Exclusive Read, Exclusive Write u CREW - Common Read, Exclusive Write u CRCW - Common Read, Common Write –All common writes must write the same thing –Highest Priority Processor wins contest u CREW is more powerful than EREW u CRCW is more powerful than CREW

Finding Max u Square Array of Processors Indexed by i,j Write True into R[i]; Read M[i] into x; Read M[j] into y; If x < y Then Write False Into R[i]; Else If y < x Then Write False Into R[j]; End If

CRCW V.S. CREW u CRCW Max runs in constant time u CREW Max runs in lg n time u CRCW cannot be any better than lg p faster than EREW

EREW V.S. CREW u Finding Roots by Shortcutting Pointers u CREW Runs in lg lg n Time u EREW Runs in lg n Time

Optimal Parallel Algorithms  NC -- The class of algorithms that run in  (log m n) time using  (n k ) processors  General Boolean Functions Cannot be Computed any Faster than  (lg n)   (lg n) is optimal for computing the sum of n integers