Complexity 19-1 Parallel Computation Complexity Andrei Bulatov

… Multiprocessor Computation Complexity 19-2 Multiprocessor Computation Obviously, several communicating processors often can solve problems faster than a single processor processor1 processor2 processor3 … processorn memory1 memory2 memory3 memoryn shared memory

Complexity 19-3 Assumptions Exchanging messages takes no time (one step in the other model) Writing to and reading from the memory takes no time and can be shared We can use as many processors as we wish (for different instances of the same problem different number of processors can be used), but this number is polynomial in the size of the input from the point of view of complexity theory, fixed number of processors makes no sense — it gives only constant speed up - we need to reveal “true” parallelism in algorithms an algorithm with unlimited number of processors can be “scaled” to an algorithm with fixed number of processors

Example Given two matrices compute their product Recall that Complexity 19-4 Example Given two matrices compute their product Recall that Straightforward computation of C can be done in Can we do better with parallel computation? Obviously, yes: use a separate processor for each of the multiplications (denote it ); then use to compute the sum for Totally, we need time

We have obtained polynomial speed-up Complexity 19-5 We have obtained polynomial speed-up This is not sufficient: changing the model of sequential computation it is possible to get some polynomial speed-up We seek for exponential drop in the time required, that is an algorithm that works in or at least

Fast Algorithm for Matrix Multiplication Complexity 19-6 Fast Algorithm for Matrix Multiplication O(log n) steps Thus, C can be computed in

Complexity 19-7 Complexity Measures We solved the Matrix Multiplication problem using processors in steps These two numbers constitute the complexity measure for parallel algorithms: number of processors time complexity As we parallelized an algorithm of time complexity the total amount of work done by a parallel algorithm cannot be less than So the minimal number of processors required is

Complexity 19-8 Brent’s Principle Our algorithm can be improved so that this theoretical bound is achieved compute in log n “shifts” using processors at each shift use the same processors to compute the first log log n parallel additions complete computation as before Since the total number of steps is at most 2 log n, this algorithm works in O(log n) Brent’s Principle If there is a parallelization, A, of a sequential algorithm that works in O(f(n)), such that the time complexity of A is g(n), then there is a parallelization, B, of the same algorithm such that B works in O(g(n)) and uses processors

Graph Reachability Reachability Complexity 19-9 Graph Reachability Instance: A finite directed graph G and vertices s and t Question: Is there a path from s to t? Reachability The natural algorithm for Reachability (breadth-first search) cannot be parallelized in an obvious way Even if many processors are arranged to process nodes from the stack, we cannot stop before a path is found that requires at least as many steps as the length of the path, that is up to n – 1 It is suspected that both searches in graphs are inherently sequential

Adjacency Matrix 1 3 4 G 2 What is ? Complexity 19-10 Adjacency Matrix 1 2 3 4 G What is ? if and only if there is a path from i to j of length 2 Therefore, to solve Reachability for all possible pairs of vertices we need to compute This can be done in parallel steps with total work