1. Complexity
19-1
Parallel Computation
Complexity
Andrei Bulatov

2. … Multiprocessor Computation
Complexity
19-2
Multiprocessor Computation
Obviously, several communicating processors often can solve problems faster than a single processor
processor1
processor2
processor3 …
processorn
memory1
memory2
memory3
memoryn
shared memory

3. Complexity
19-3
Assumptions
Exchanging messages takes no time (one step in the other model)
Writing to and reading from the memory takes no time and can be shared
We can use as many processors as we wish (for different instances of
the same problem different number of processors can be used), but this
number is polynomial in the size of the input
from the point of view of complexity theory, fixed number of processors
makes no sense — it gives only constant speed up
- we need to reveal “true” parallelism in algorithms
an algorithm with unlimited number of processors can be “scaled” to an
algorithm with fixed number of processors

4. Example Given two matrices compute their product Recall that
Complexity
19-4
Example
Given two matrices
compute their product
Recall that
Straightforward computation of C can be done in
Can we do better with parallel computation?
Obviously, yes: use a separate processor for each of the multiplications (denote it );
then use to compute the sum for
Totally, we need time

5. We have obtained polynomial speed-up
Complexity
19-5
We have obtained polynomial speed-up
This is not sufficient: changing the model of sequential computation it is possible to get some polynomial speed-up
We seek for exponential drop in the time required, that is an algorithm that works in or at least

6. Fast Algorithm for Matrix Multiplication
Complexity
19-6
Fast Algorithm for Matrix Multiplication
O(log n) steps
Thus, C can be computed in

7. Complexity
19-7
Complexity Measures
We solved the Matrix Multiplication problem using processors in
steps
These two numbers constitute the complexity measure for parallel algorithms:
number of processors
time complexity
As we parallelized an algorithm of time complexity the total amount of work done by a parallel algorithm cannot be less than
So the minimal number of processors required is

8. Complexity
19-8
Brent’s Principle
Our algorithm can be improved so that this theoretical bound is achieved
compute in log n “shifts” using processors at each shift
use the same processors to compute the first log log n
parallel additions
complete computation as before
Since the total number of steps is at most 2 log n, this algorithm works in
O(log n)
Brent’s Principle
If there is a parallelization, A, of a sequential algorithm that
works in O(f(n)), such that the time complexity of A is
g(n), then there is a parallelization, B, of the same algorithm
such that B works in O(g(n)) and uses processors

9. Graph Reachability Reachability
Complexity
19-9
Graph Reachability
Instance: A finite directed graph G and vertices s and t
Question: Is there a path from s to t?
Reachability
The natural algorithm for Reachability (breadth-first search) cannot be parallelized in an obvious way
Even if many processors are arranged to process nodes from the stack, we cannot stop before a path is found that requires at least as many steps as the length of the path, that is up to n – 1
It is suspected that both searches in graphs are inherently sequential

10. Adjacency Matrix 1 3 4 G 2 What is ?
Complexity
19-10
Adjacency Matrix 1 2 3 4 G
What is ?
if and only if there is a path from i to j of length 2
Therefore, to solve Reachability for all possible pairs of vertices we need to compute
This can be done in parallel steps with total work