CS 345: Chapter 10 Parallelism, Concurrency, and Alternative Models Or, Getting Lots of Stuff Done at Once

Parallelism is the use of many processors to solve a problem. When we want to ask the question, “Will parallelism result in some intractable problems becoming tractable?” we need to distinguish between fixed parallelism and expanding parallelism.

Fixed Parallelism: The number of processors remains fixed as the problem size grows. Expanding Parallelism: The number of processors available grows as the problem size grows. Since the number of processors available affects the running time of a parallel algorithm, how do we balance the two?

Product Complexity: Time x Size. See Figure 10.2, page 262. Note that since any parallel algorithm can be serialized on a single processor, the best product complexity cannot be lower than the lower bound of the problem’s sequential complexity.

Among the difficulties of parallel algorithms is “How should processors communicate?” There are two basic approaches. –Shared Memory –Fixed Networks

Shared Memory These types of machines are called Multi ‑ Processors. Is the shared memory only for reading values or can it be used to write values? If the shared memory is for writing, then there must be some method for resolving conflicts.

Fixed-Network Machines These types of computers are called Multi ‑ Computers. A processor is connected to a fixed number of neighbors, and each processing element has its own private memory. There are many types of network designs: mesh/torus, fat trees, hypercubes, and others based on graph theory.

Some networks are designed for specific problems. Others are for general purpose, but are better at some algorithms than others. Some issues with large multi-computers –What communication model is used? Store and forward Worm-hole routing Virtual cut-through –Deadlock and Livelock –Fault-tolerance.

Systolic Networks One type of network for parallel processing is a systolic network. See figure 10.5 on page 266. This is similar to an assembly line. Each processor performs a set task on some data and passes it on to its neighbor. In the example, there is an n x m matrix. A sequential algorithm takes O( n x m ), but a systolic network takes O( n + m ).

No. Since any parallel algorithm can be serialized, a parallel algorithm cannot solve an undecidable or non-computable problem. Can a parallel algorithm solve an undecidable problem?

Can a parallel algorithm turn an intractable problem into a tractable problem? Many problems in NP, including some NP- Complete problems, have been shown to have a polynomial time parallel solution. However –The number of processors required for the solution is exponential.

–NP-complete problems are not known to be intractable, so this does not necessarily mean that parallelism can remove inherent intractability. –It is not clear that a parallel algorithm that uses only a polynomial number of instructions but requires an exponential number of processors can actually run in polynomial time on a real computer. Thus, the question of whether a parallel algorithm can remove inherent intractability, is still an open question.

The Parallel Computation Thesis Part of this thesis is the claim that parallel time is the same as sequential memory. If P can be solved sequentially using S space for inputs of length N, then it can be solved in parallel time that is no worse than polynomial time in S.

Alternately, if P can be solved in parallel time T with inputs of length N, it can be solved sequentially using memory bounded by a polynomial in T.

So, any algorithm solvable by a sequential algorithm in polynomial space can be solved by a parallel algorithm in polynomial time. That is: Sequential-PSpace = Parallel-PTime

Thus the question of whether there are intractable problems that become tractable when using parallelism comes down to the question: Does PSpace contain intractable problems? This, like the question, “Does P=NP?”, is still an open question.

The Class NC (Nick’s Class) In general, a polynomial time parallel algorithm cannot claim to be tractable since it may require an exponential number of processors. A problem is in NC if –It runs in polylogrithmic time. –It requires only a polynomial number of processors.

All problems in NC are also in P, but it is not known if the converse is true. Most researchers believe that the two classes are not the same. Thus, we have: NC  P  NP  PSpace

Conjecture 1 There are problems solvable in reasonable sequential space – reasonable parallel time that cannot be solved in reasonable sequential time, even using non- determinism.

Conjecture 2 There are problems solvable in reasonable sequential time only if non-determinism is used.

Conjecture 3 There are problems solvable in reasonable sequential space, but not in very fast parallel time using reasonable hardware size.