Data Parallelism Task Parallel Library (TPL) The use of lambdas Map-Reduce Pattern FEN 20141UCN Teknologi/act2learn

Multi Cores and Multithreading Today it seems that Moore’s law doesn’t apply to processor speed anymore. Recently we haven’t got more CPU speed. Instead we got more cores. Applications should take advantage of this by being multithreaded. Demo: demos\parallel1demos\parallel1 FEN 2014UCN Teknologi/act2learn2

Parallel For-loop Note: – Parallel.For(..) is actually a higher order function taking a lambda (the loop body) as argument. FEN 2014UCN Teknologi/act2learn3

How much can we gain by many cores? Amdahl's law ( Shows maximum speedup factor relative to number of processors. “Parallel portion” is the percentage of the code that can be parallelized. How many cores do you have? (Remember to count your graphics card processors ) FEN 2014UCN Teknologi/act2learn4

From Founder of Amdahl Corporation, now a part of Fujitsu: FEN 2014UCN Teknologi/act2learn5

Number of cores – number of threads? View the task manager while you experiment with noOfThreads Too few or too many threads? demos\parallel2 FEN 2014UCN Teknologi/act2learn6

Parallelism and higher order functions (lambdas) This constructor is actually a higher order function receiving this function as argument. FEN 2014UCN Teknologi/act2learn7

Another example where Parallel.For doesn’t work Summing an array: Parallel.For does it fast, but gets it wrong. What’s the problem? FEN 2014UCN Teknologi/act2learn8 demos\CS-code\SummingParallel_2

It can be fixed: int sum = 0; object lockObj = new object(); Parallel.ForEach( a, () => 0, (x, loopState, partialSum) => { SlowDown(slowFactor); return x + partialSum; }, (localPartialSum) => { lock (lockObj) { sum = sum + localPartialSum; } ); FEN 2014UCN Teknologi/act2learn9 demos\SummingParallelV3 But this is not a simple higher order function !

Parallel Invoke Easy to start many concurrent actions using Parallel.Invoke.Parallel.Invoke Invoke is a higher order function taking an array of Action delegates as argument. FEN 2014UCN Teknologi/act2learn10

Map Reduce Pattern FEN 2014UCN Teknologi/act2learn11 The Map-Reduce pattern from functional programming makes often handling parallelism easier. Among others, it’s used in processing large datasets in a distributed and parallel environment (Google, for instance).

Map Reduce Pattern FEN 2014UCN Teknologi/act2learn12 The idea is to make it easier to parallelize an algorithm (aggregation, for instance) by splitting it into a map phase and a reduce phase. The map-phase doesn’t change the data structure, but creates a new one prepared for the reduce phase. Since the map-phase doesn’t change state it’s easier to parallelize.

Map Reduce Pattern FEN 2014UCN Teknologi/act2learn13 Let’s assume that we have some huge collection of objects and want to find the number of objects with some given property: Map-phase: Create a new collection to contain 0s and 1s, iterate through the collection and add 1 if the current object has the property, add 0 otherwise. – Since the original container isn’t changed and updates in new container take place in different positions this phase is easily parallelized. Reduce-phase: count the number of 1s. In this phase synchronization may be necessary. Lots of objects Map … Reduce Result (aggregation)

Map Reduce Pattern FEN 2014UCN Teknologi/act2learn14 In the container: – Map returns a new Container Func of op Func of f

Map Reduce Pattern FEN 2014UCN Teknologi/act2learn15 Call with lambdas: demos\ParallelMapReduce Two input parameters (Func of op) Recall: a lambda can be a block of code (‘{---}‘)

Map Reduce Pattern FEN 2014UCN Teknologi/act2learn16 Map can be parallelized easily: demos\CS-code\ParallelMapReduce Just use Parallel.For(-)

Many cores cannot beat a good algorithm! FEN 2014UCN Teknologi/act2learn17 \ParallelSort \MergeSort \ParallelSort2(SeqBubble) Speed Up:

Sorting n: no of elements c: no of cores – Sequential bubble sort: O(n 2 ) – Parallel bubble sort: O((n/c) 2 ) – If the problem is O(n 2 ), then the speedup is O(c 2 ) – Mergesort: O(n*log 2 n) FEN 2014UCN Teknologi/act2learn18

Many cores cannot beat a good algorithm! …and it gets worse as n increases: FEN 2014UCN Teknologi/act2learn19 \ParallelSort \MergeSort

…and it gets worse and worse as n increases: FEN 2014UCN Teknologi/act2learn20 And we save a lot of power and help to save the Earth’s climat !

Final remarks In the future, we, as programmers won’t be saved by Moore’s Law. We need to use these many cores in our applications. And do it intelligently. FEN 2014UCN Teknologi/act2learn21