1. Parallel Processing
I’ve gotta spend at least 10 hours studying for the IT 344 final!
I’m going to study with 9 friends… we’ll be done in an hour.

2. Next up: TIPS Mega- = 106, Giga- = 109, Tera- = 1012, Peta- = 1015
BOPS, anyone?
Light travels about 1 ft / 10-9 secs in free space.
A Tera-Hertz uniprocessor could have no clock-to-clock path longer than 300 microns…
We already know of problems that require greater than a TIP (Simulations of weather, weapons, brains)

3. Solution: Parallelism
Pipelining – reasonable for a small number of stages (5-10), after that bypassing and stalls become unmanageable.
Superscalar – replicate data paths and design control logic to discover parallelism in traditional programs.
Explicit parallelism – must learn how to write programs that run on multiple CPUs.

4. Pipelining

5. Superscalar – How far can it go?
Multiple functional units (ALUs, Addr, Floating point, etc.)
Instruction dispatch
Dynamic scheduling
Pipelines
Speculative execution

6. Explicit Parallelism Distributed Parallel Transaction-oriented
Geographically dispersed locations
E.g.
Parallel
Single goal computing
Computing intense and/or data-intense
High-speed data exchange
Often on custom hardware
E.g. Geochemical surveys

7. Challenges
For distributed processing, parallelism is given and usually cannot easily change. Programming is relatively easy.
For parallel processing, the programmer defines parallelism by partitioning the serial program(s). Parallel programming in general is more difficult than transaction applications.

8. Other vocabulary Decomposition Course-grain Fine-grain
The way that a program can be broken up for parallel processing
Course-grain
Breaks into big chunks (fewer processors)
SMP
Distributed (often)
Fine-grain
Breaks into small chunks (more processors)
Image processing

9. Inter-processor communications
Loosely-coupled
Tightly-coupled
Custom supercomputers
Distributed processors
Beowulf clusters

10. More Terminology
SIMD (Single Instruction Multiple Data)
MIMD (Multiple Instruction Multiple Data)
MISD (Pipeline)

11. SIMD Same instruction executed in multiple units, on different data I
Examples: Vector processors, AltiVec D1 I D2 I D3 I D4 I

12. MIMD Each unit does own instruction on own text
Examples: Mercury, Beowulf, etc. I1 I2 I3 I4 D1 D2 D3 D4

13. MISD (pipeline) D4 D3 D2 D1 I1 I2 I3 I4

14. Distributed Programming Tools
C/C++ with TCP/IP
Perl with TCP/IP
Java
Corba
ASP
.Net

15. Parallel Programming Tools
PVM
MPI
Synergy
Others (proprietary hardware)

16. Parallel Programming Difficulties
Program partition and allocation
Data partition and allocation
Program(process) synchronization
Data access mutual exclusion
Dependencies
Process(or) failures
Scalability…

17. Software techniques
Shared Memory Buffers — Areas of memory that any node can read or write
Sockets — Provide full-duplex message passing between processes.
Semaphores and Spinlocks — Provide locking and synchronization functions
Mailbox Interrupts — Provide an interrupt-driven communication mechanism
Direct Memory Access — Provides asynchronous shared memory bufferI/O.

18. Hardware configurations – Interconnects and Memory

19. Interconnects

20. Crossbar

21. Mesh

22. Interconnects

23. What it really looks like
Note: this computer would rank well on

24. Summary Prospects for future CPU architectures:
Pipelining - Well understood, but mined-out
Superscalar - Nearing its practical limits
SIMD - Limited use for special applications
VLIW - Returns controls to S/W. The future?
Prospects for future Computer System architectures:
SMP - Limited scalability. Harder than it appears.
MIMD/message-passing - It’s been the future for over 20 years now. How to program?