The Truth About Parallel Computing: Fantasy versus Reality William M. Jones, PhD Computer Science Department Coastal Carolina University

What is Parallel Computing ? Simply put: Parallel computing is the simultaneous use of multiple compute resources to solve a computational problem. … sounds simple …

What typifies these “computational problems” ? ● Can be broken apart into discrete pieces of work that can be solved simultaneously ● Can be solved in less time with multiple compute resources than with a single compute resource … however … ● Problem decomposition can be exceedingly complex ● Ultimate performance depends on a rather large number of interacting factors

Some Typical Architectures: Shared Memory ● Multiple CPU's with global address space ● Each CPU can “work” on a part of the problem as the same time ● Data sharing takes place in main-memory

Some Typical Architectures: Distributed Memory ● Multiple computers with local address space ● Each computer can “work” on a part of the problem as the same time ● Data sharing takes place by sending messages across the network

Example Programming Model Data Parallel ● Large data-set ● Partitioned across data ● Each task works on it's part of the data ● Suppose data needs to be exchanged at runtime ● Communication may be necessary

Program Granularity ● Communication / Computation Ratio ● Coarse-grain ● Fine-grain ● Communication can be a dramatic bottleneck ● Optimization

More Bad News: Theoretical Limits to Parallel Program Performance

Amdahl's Law, Continued

Conclusion ● Parallel computing can help ● Problem decomposition is often difficult ● Often, data exchange is necessary, thus communication is necessary ● Communication is often a bottleneck ● Even if no communication is necessary, ultimate performance is limited to fraction of program that is parallelizable ● Questions ?