1. Distributed and Parallel Processing George Wells

2. Course Overview Introduction and Background –Terminology –Hardware issues Parallel algorithm patterns Parallel Processing –Threads, Interprocess communication, CSP, etc. Distributed Processing –Remote Procedure/Method Calls, Virtual Shared Memory, Message Oriented Middleware, Grid Computing, etc. Formal specification –CSP

3. Levels of Abstraction Liu identifies a number of paradigms that form a hierarchy of abstractions –Useful for classifying parallel/distributed processing methods Object “spaces” Net. services, ORBs, mobile agents RPC/RMI Client-server, peer-to-peer Message passing Abstraction Level

4. Vitally Important Topic Multicore processors –Increasing numbers of cores –Need for parallel software Increasing emphasis on grid/cloud computing

5. Processor Design Historically, performance improvements came mainly from two areas: – Architecture Doing clever things with sequential instruction streams (pipelining, speculative execution, branch prediction, etc.) – Technology Primarily clock speed Both of these have plateaued

6. Power and Frequency Power vs. Frequency Curve for Single Core Architecture 9 59 109 159 209 259 309 359 00.20.40.60.811.21.41.61.822.22.42.62.833.23.4 Frequency (GHz) Power (w) Dropping Frequency = Large Drop Power

7. Why Distributed and Parallel ? Logical reasonsPhysical reasons Closing the semantic gap between problem domain and software solution Speed Fault tolerance Spatial convenience

8. Revision of undergrad architecture CPU MemoryI/O

9. Control unit Processing unit CPU MemoryI/O The Von Neumann machine machine or RAM (Random Access Machine)‏

10. Multiple PU’s (many modern processors)‏ Control unit Processing unit MemoryI/O

11. Multiple CPUs/Cores (Multiprocessor with shared memory)‏ Control unit Processing unit MemoryI/O Control unit Processing unit Control unit Processing unit

12. The Sun UltraSPARC T2 Processor 8 cores 503 million transistors 18.5mm

13. Multiple Computers Control unit Processing unit MemoryI/O Control unit Processing unit Control unit Processing unit MemoryI/OMemoryI/O

14. Cross-bar Arrangement Control unit Processing unit MemoryI/O Control unit Processing unit Control unit Processing unit MemoryI/OMemoryI/O

15. Much of distributed and parallel programming structures are about managing complexity

16. Terminology Concurrency and Parallelism –parallel if at some instant in time both are actually executing –concurrent if they have the potential for executing in parallel

17. Terminology Parallel and Distributed –parallel usually refers to tightly-coupled systems for fine-grained problems e.g. Finite-element analysis –distributed usually refers to loosely-coupled systems for coarse-grained problems e.g. Distributed search applications

18. Terminology Processes, processors, and tasks –task is an operation or a set of operations to be performed –process is an instance of the task –thread is a unit of control within a process –processors are those things which execute processes Comment: talk about memory/addr space issues for process -vs- thread

19. Terminology Processes, processors, and tasks –time-slicing between several processes/threads gives rise to the concept of a virtual machine or abstract processor –a process’ or thread’s life-history can be defined in terms of three primary states: executing, executable, and suspended