1. Parallel Patterns Reduce & Scan
6/16/2010
Parallel Patterns - Reduce & Scan
Parallel Patterns Reduce & Scan

2. Programming Patterns For Parallelism
6/16/2010
Parallel Patterns - Reduce & Scan
Programming Patterns For Parallelism
Some patterns repeat in many different contexts
e.g. Search an element in an array
Identifying such patterns important
Solve a problem once and reuse the solution
Split a hard problem into individual problems
Helps define interfaces

3. We Have Already Seen Some Patterns
6/16/2010
Parallel Patterns - Reduce & Scan
We Have Already Seen Some Patterns

4. We Have Already Seen Some Patterns
6/16/2010
Parallel Patterns - Reduce & Scan
We Have Already Seen Some Patterns
Divide and Conquer
Split a problem into n sub problems
Recursively solve the sub problems
And merge the solution
Data Parallelism
Apply the same function to all elements in a collection, array
Parallel.For, Parallel.ForEach
Also called as “map” in functional programming

5. Parallel Patterns - Reduce & Scan
6/16/2010
Parallel Patterns - Reduce & Scan
Map
Given a function f : (A) => B
A collection a: A[]
Generates a collection b: B[], where B[i] = f( A[i] )
Parallel.For, Paralle.ForEach
Where each loop iteration is independent A f B

6. Parallel Patterns - Reduce & Scan
6/16/2010
Parallel Patterns - Reduce & Scan
Reduce And Scan
In practice, parallel loops have to work together to generate an answer
Reduce and Scan patterns capture common cases of processing results of Map

7. Parallel Patterns - Reduce & Scan
6/16/2010
Parallel Patterns - Reduce & Scan
Reduce And Scan
In practice, parallel loops have to work together to generate an answer
Reduce and Scan patterns capture common cases of processing results of Map
Note: Map and Reduce are similar to but not the same as MapReduce
MapReduce is a framework for distributed computing

8. Parallel Patterns - Reduce & Scan
6/16/2010
Parallel Patterns - Reduce & Scan
Reduce
Given a function f: (A, B) => B
A collection a: A[]
An initial value b0: B
Generate a final value b: B
Where b = f(A[n-1], … f(A[1], f(A[0], b0)) ) A f b b0

9. Parallel Patterns - Reduce & Scan
6/16/2010
Parallel Patterns - Reduce & Scan
Reduce
Given a function f: (A, B) => B
A collection a: A[]
An initial value b0: B
Generate a final value b: B
Where b = f(A[n-1], … f(A[1], f(A[0], b0)) )
Only consider where A and B are the same type A f b b0

10. Parallel Patterns - Reduce & Scan
6/16/2010
Parallel Patterns - Reduce & Scan
Reduce
B acc = b_0;
for( i = 0; i < n; i++ ) {
acc = f( a[i], acc ); }
b = acc; A f b b0

11. Associativity of the Reduce function
6/16/2010
Parallel Patterns - Reduce & Scan
Associativity of the Reduce function
Reduce is parallelizable if f is associative
f(a, f(b, c)) = f(f(a,b), c)
E.g. Addition : (a + b) + c = a + (b + c)
Where + is integer addition (with modulo arithmetic)
But not when + is floating point addition

12. Associativity of the Reduce function
6/16/2010
Parallel Patterns - Reduce & Scan
Associativity of the Reduce function
Reduce is parallelizable if f is associative
f(a, f(b, c)) = f(f(a,b), c)
E.g. Addition : (a + b) + c = a + (b + c)
Where + is integer addition (with modulo arithmetic)
But not when + is floating point addition
Max, min, multiply, …
Set union, intersection,

13. We can use Divide and Conquer
6/16/2010
Parallel Patterns - Reduce & Scan
We can use Divide and Conquer
Reduce(f, A[1…n], b_0)
= f ( Reduce(f, A[1..n/2], b_0),
Reduce(f, A[n/2+1…n], I) )
where I is the identity element of f A f f f f f f f f b0 I f b

14. Implementation Optimizations
6/16/2010
Parallel Patterns - Reduce & Scan
Implementation Optimizations
Switch to sequential Reduce for the base k elements
Do k way splits instead of two way splits
Maintain a thread-local accumulated value
A task updates the value of the thread it executes in

15. Implementation Optimizations
6/16/2010
Parallel Patterns - Reduce & Scan
Implementation Optimizations
Switch to sequential Reduce for the base k elements
Do k way splits instead of two way splits
Maintain a thread-local accumulated value
A task updates the value of the thread it executes in
Requires that the reduce function is also commutative
f(a, b) = f(b, a)

16. Implementation Optimizations
6/16/2010
Parallel Patterns - Reduce & Scan
Implementation Optimizations
Switch to sequential Reduce for the base k elements
Do k way splits instead of two way splits
Maintain a thread-local accumulated value
A task updates the value of the thread it executes in
Requires that the reduce function is also commutative
f(a, b) = f(b, a)
Thread local values are then merged in a separate pass

17. Parallel Patterns - Reduce & Scan
6/16/2010
Parallel Patterns - Reduce & Scan
Scan
Given a function f: (A, B) => B
A collection a: A[]
An initial value b0: B
Generate a collection b: B[]
Where b[i] = f(A[i-1], … f(A[1], f(A[0], b0)) ) A f b0

18. Parallel Patterns - Reduce & Scan
6/16/2010
Parallel Patterns - Reduce & Scan
Scan
B acc = b_0;
for( i = 0; i < n; i++ ) {
acc = f( a[i], acc ); } A f b0

19. Scan is Efficiently Parallelizable
6/16/2010
Parallel Patterns - Reduce & Scan
Scan is Efficiently Parallelizable
When f is associative

20. Scan is Efficiently Parallelizable
6/16/2010
Parallel Patterns - Reduce & Scan
Scan is Efficiently Parallelizable
When f is associative
Scan(f, A[1..n], b_0) = Scan(f, A[1..n/2], b_0),
Scan(f, A[n/2+1…n], ____) A f f f f f f f f b0 ?

21. Scan is Efficiently Parallelizable
6/16/2010
Parallel Patterns - Reduce & Scan
Scan is Efficiently Parallelizable
When f is associative
Scan(f, A[1..n], b_0) = Scan(f, A[1..n/2], b_0),
Scan(f, A[n/2+1…n],
Reduce(f, A[1..n/2], b_0)) A f f f f f f f f b0 ?

22. Scan is useful in many places
6/16/2010
Parallel Patterns - Reduce & Scan
Scan is useful in many places
Radix Sort
Ray Tracing …

23. Scan is useful in many places
6/16/2010
Parallel Patterns - Reduce & Scan
Scan is useful in many places
Radix Sort (  )
Ray Tracing …

24. Computing Line of Sight
6/16/2010
Parallel Patterns - Reduce & Scan
Computing Line of Sight
Given x1, … xn with altitudes a[1],…a[n]
Which of the points are visible from x0

25. Computing Line of Sight
6/16/2010
Parallel Patterns - Reduce & Scan
Computing Line of Sight
Given x0, … xn with altitudes alt[0],…alt[n]
Which of the points are visible from x0
angle[i] = arctan( (alt[i] – alt[0]) / i )
xi is visible from x0 if all points between them have lesser angle than angle[i]

26. Parallel Patterns - Reduce & Scan
6/16/2010
Parallel Patterns - Reduce & Scan
Solution

27. Parallel Patterns - Reduce & Scan
6/16/2010
Parallel Patterns - Reduce & Scan
Radix Sort
5 = 101
7 = 111
2 = 010
4 = 100
3 = 011
1 = 001

28. Parallel Patterns - Reduce & Scan
6/16/2010
Parallel Patterns - Reduce & Scan
Radix Sort
5 = 101
7 = 111
2 = 010
4 = 100
3 = 011
1 = 001
2 = 010
4 = 100
5 = 101
7 = 111
3 = 011
1 = 001

29. Parallel Patterns - Reduce & Scan
6/16/2010
Parallel Patterns - Reduce & Scan
Radix Sort
5 = 101
7 = 111
2 = 010
4 = 100
3 = 011
1 = 001
2 = 010
4 = 100
5 = 101
7 = 111
3 = 011
1 = 001
4 = 100
5 = 101
1 = 001
2 = 010
7 = 111
3 = 011

30. Parallel Patterns - Reduce & Scan
6/16/2010
Parallel Patterns - Reduce & Scan
Radix Sort
5 = 101
7 = 111
2 = 010
4 = 100
3 = 011
1 = 001
2 = 010
4 = 100
5 = 101
7 = 111
3 = 011
1 = 001
4 = 100
5 = 101
1 = 001
2 = 010
7 = 111
3 = 011
1 = 001
2 = 010
3 = 011
4 = 100
5 = 101
7 = 111

31. Parallel Patterns - Reduce & Scan
6/16/2010
Parallel Patterns - Reduce & Scan
Basic Primitive: Pack
Given an array A and an array F of flags
A = [ ]
F = [ ]
Pack all elements with flag = 0 before elements with flag = 1
A’ = [ ]

32. Parallel Patterns - Reduce & Scan
6/16/2010
Parallel Patterns - Reduce & Scan
Solution

33. Other Applications of Scan
6/16/2010
Parallel Patterns - Reduce & Scan
Other Applications of Scan
Radix Sort
Computing Line of Sight
Adding multi-precision numbers
Quick Sort
To search for regular expressions
Parallel grep …

34. Parallel Patterns - Reduce & Scan
6/16/2010
Parallel Patterns - Reduce & Scan
High Level Points
Minimize dependence between parallel loops
Unintended dependences = data races
Next lecture
Carefully analyze remaining dependences
Use Reduce and Scan patterns where applicable