Parallel Patterns Reduce & Scan 6/16/2010 Parallel Patterns - Reduce & Scan Parallel Patterns Reduce & Scan

Programming Patterns For Parallelism 6/16/2010 Parallel Patterns - Reduce & Scan Programming Patterns For Parallelism Some patterns repeat in many different contexts e.g. Search an element in an array Identifying such patterns important Solve a problem once and reuse the solution Split a hard problem into individual problems Helps define interfaces

We Have Already Seen Some Patterns 6/16/2010 Parallel Patterns - Reduce & Scan We Have Already Seen Some Patterns

We Have Already Seen Some Patterns 6/16/2010 Parallel Patterns - Reduce & Scan We Have Already Seen Some Patterns Divide and Conquer Split a problem into n sub problems Recursively solve the sub problems And merge the solution Data Parallelism Apply the same function to all elements in a collection, array Parallel.For, Parallel.ForEach Also called as “map” in functional programming

Parallel Patterns - Reduce & Scan 6/16/2010 Parallel Patterns - Reduce & Scan Map Given a function f : (A) => B A collection a: A[] Generates a collection b: B[], where B[i] = f( A[i] ) Parallel.For, Paralle.ForEach Where each loop iteration is independent A f B

Parallel Patterns - Reduce & Scan 6/16/2010 Parallel Patterns - Reduce & Scan Reduce And Scan In practice, parallel loops have to work together to generate an answer Reduce and Scan patterns capture common cases of processing results of Map

Parallel Patterns - Reduce & Scan 6/16/2010 Parallel Patterns - Reduce & Scan Reduce And Scan In practice, parallel loops have to work together to generate an answer Reduce and Scan patterns capture common cases of processing results of Map Note: Map and Reduce are similar to but not the same as MapReduce MapReduce is a framework for distributed computing

Parallel Patterns - Reduce & Scan 6/16/2010 Parallel Patterns - Reduce & Scan Reduce Given a function f: (A, B) => B A collection a: A[] An initial value b0: B Generate a final value b: B Where b = f(A[n-1], … f(A[1], f(A[0], b0)) ) A f b b0

Parallel Patterns - Reduce & Scan 6/16/2010 Parallel Patterns - Reduce & Scan Reduce Given a function f: (A, B) => B A collection a: A[] An initial value b0: B Generate a final value b: B Where b = f(A[n-1], … f(A[1], f(A[0], b0)) ) Only consider where A and B are the same type A f b b0

Parallel Patterns - Reduce & Scan 6/16/2010 Parallel Patterns - Reduce & Scan Reduce B acc = b_0; for( i = 0; i < n; i++ ) { acc = f( a[i], acc ); } b = acc; A f b b0

Associativity of the Reduce function 6/16/2010 Parallel Patterns - Reduce & Scan Associativity of the Reduce function Reduce is parallelizable if f is associative f(a, f(b, c)) = f(f(a,b), c) E.g. Addition : (a + b) + c = a + (b + c) Where + is integer addition (with modulo arithmetic) But not when + is floating point addition

Associativity of the Reduce function 6/16/2010 Parallel Patterns - Reduce & Scan Associativity of the Reduce function Reduce is parallelizable if f is associative f(a, f(b, c)) = f(f(a,b), c) E.g. Addition : (a + b) + c = a + (b + c) Where + is integer addition (with modulo arithmetic) But not when + is floating point addition Max, min, multiply, … Set union, intersection,

We can use Divide and Conquer 6/16/2010 Parallel Patterns - Reduce & Scan We can use Divide and Conquer Reduce(f, A[1…n], b_0) = f ( Reduce(f, A[1..n/2], b_0), Reduce(f, A[n/2+1…n], I) ) where I is the identity element of f A f f f f f f f f b0 I f b

Implementation Optimizations 6/16/2010 Parallel Patterns - Reduce & Scan Implementation Optimizations Switch to sequential Reduce for the base k elements Do k way splits instead of two way splits Maintain a thread-local accumulated value A task updates the value of the thread it executes in

Implementation Optimizations 6/16/2010 Parallel Patterns - Reduce & Scan Implementation Optimizations Switch to sequential Reduce for the base k elements Do k way splits instead of two way splits Maintain a thread-local accumulated value A task updates the value of the thread it executes in Requires that the reduce function is also commutative f(a, b) = f(b, a)

Implementation Optimizations 6/16/2010 Parallel Patterns - Reduce & Scan Implementation Optimizations Switch to sequential Reduce for the base k elements Do k way splits instead of two way splits Maintain a thread-local accumulated value A task updates the value of the thread it executes in Requires that the reduce function is also commutative f(a, b) = f(b, a) Thread local values are then merged in a separate pass

Parallel Patterns - Reduce & Scan 6/16/2010 Parallel Patterns - Reduce & Scan Scan Given a function f: (A, B) => B A collection a: A[] An initial value b0: B Generate a collection b: B[] Where b[i] = f(A[i-1], … f(A[1], f(A[0], b0)) ) A f b0

Parallel Patterns - Reduce & Scan 6/16/2010 Parallel Patterns - Reduce & Scan Scan B acc = b_0; for( i = 0; i < n; i++ ) { acc = f( a[i], acc ); } A f b0

Scan is Efficiently Parallelizable 6/16/2010 Parallel Patterns - Reduce & Scan Scan is Efficiently Parallelizable When f is associative

Scan is Efficiently Parallelizable 6/16/2010 Parallel Patterns - Reduce & Scan Scan is Efficiently Parallelizable When f is associative Scan(f, A[1..n], b_0) = Scan(f, A[1..n/2], b_0), Scan(f, A[n/2+1…n], ____) A f f f f f f f f b0 ?

Scan is Efficiently Parallelizable 6/16/2010 Parallel Patterns - Reduce & Scan Scan is Efficiently Parallelizable When f is associative Scan(f, A[1..n], b_0) = Scan(f, A[1..n/2], b_0), Scan(f, A[n/2+1…n], Reduce(f, A[1..n/2], b_0)) A f f f f f f f f b0 ?

Scan is useful in many places 6/16/2010 Parallel Patterns - Reduce & Scan Scan is useful in many places Radix Sort Ray Tracing …

Scan is useful in many places 6/16/2010 Parallel Patterns - Reduce & Scan Scan is useful in many places Radix Sort (  ) Ray Tracing …

Computing Line of Sight 6/16/2010 Parallel Patterns - Reduce & Scan Computing Line of Sight Given x1, … xn with altitudes a[1],…a[n] Which of the points are visible from x0

Computing Line of Sight 6/16/2010 Parallel Patterns - Reduce & Scan Computing Line of Sight Given x0, … xn with altitudes alt[0],…alt[n] Which of the points are visible from x0 angle[i] = arctan( (alt[i] – alt[0]) / i ) xi is visible from x0 if all points between them have lesser angle than angle[i]

Parallel Patterns - Reduce & Scan 6/16/2010 Parallel Patterns - Reduce & Scan Solution

Parallel Patterns - Reduce & Scan 6/16/2010 Parallel Patterns - Reduce & Scan Radix Sort 5 = 101 7 = 111 2 = 010 4 = 100 3 = 011 1 = 001

Parallel Patterns - Reduce & Scan 6/16/2010 Parallel Patterns - Reduce & Scan Radix Sort 5 = 101 7 = 111 2 = 010 4 = 100 3 = 011 1 = 001 2 = 010 4 = 100 5 = 101 7 = 111 3 = 011 1 = 001

Parallel Patterns - Reduce & Scan 6/16/2010 Parallel Patterns - Reduce & Scan Radix Sort 5 = 101 7 = 111 2 = 010 4 = 100 3 = 011 1 = 001 2 = 010 4 = 100 5 = 101 7 = 111 3 = 011 1 = 001 4 = 100 5 = 101 1 = 001 2 = 010 7 = 111 3 = 011

Parallel Patterns - Reduce & Scan 6/16/2010 Parallel Patterns - Reduce & Scan Radix Sort 5 = 101 7 = 111 2 = 010 4 = 100 3 = 011 1 = 001 2 = 010 4 = 100 5 = 101 7 = 111 3 = 011 1 = 001 4 = 100 5 = 101 1 = 001 2 = 010 7 = 111 3 = 011 1 = 001 2 = 010 3 = 011 4 = 100 5 = 101 7 = 111

Parallel Patterns - Reduce & Scan 6/16/2010 Parallel Patterns - Reduce & Scan Basic Primitive: Pack Given an array A and an array F of flags A = [5 7 2 4 5 3 1] F = [1 1 0 0 1 1 1] Pack all elements with flag = 0 before elements with flag = 1 A’ = [2 4 5 7 5 3 1]

Parallel Patterns - Reduce & Scan 6/16/2010 Parallel Patterns - Reduce & Scan Solution

Other Applications of Scan 6/16/2010 Parallel Patterns - Reduce & Scan Other Applications of Scan Radix Sort Computing Line of Sight Adding multi-precision numbers Quick Sort To search for regular expressions Parallel grep …

Parallel Patterns - Reduce & Scan 6/16/2010 Parallel Patterns - Reduce & Scan High Level Points Minimize dependence between parallel loops Unintended dependences = data races Next lecture Carefully analyze remaining dependences Use Reduce and Scan patterns where applicable