1. Parallel Processing: Architecture Overview
WW Grid
Rajkumar Buyya
Grid Computing and Distributed Systems (GRIDS) Lab. The University of Melbourne Melbourne, Australia

2. Serial Vs. Parallel
COUNTER 2
COUNTER
COUNTER 1 Q
Please

3. Overview of the Talk Introduction Why Parallel Processing ?
Parallel System H/W Architecture
Parallel Operating Systems

4. Multi-Processor Computing System
Computing Elements
Applications
Programming paradigms P .
Microkernel
Multi-Processor Computing System
Threads Interface
Operating System
Hardware
Process
Processor
Thread P

5. Two Eras of Computing Architectures System Software/Compiler
Applications
P.S.Es
System Software
Sequential
Era
Parallel
Era
Commercialization
R & D Commodity

6. History of Parallel Processing
The notion of parallel processing can be traced to a tablet dated around 100 BC.
Tablet has 3 calculating positions capable of operating simultaneously.
From this we can infer that:
They were aimed at “speed” or “reliability”.

7. Motivating Factors
Just as we learned to fly, not by constructing a machine that flaps its wings like birds, but by applying aerodynamics principles demonstrated by the nature...
Similarly parallel processing has been modeled after those of biological species.
Aggregated speed with which complex calculations carried out by (billions of) neurons demonstrate feasibility of PP.
Individual neuron response speed is slow (ms) –

8. Why Parallel Processing?
Computation requirements are ever increasing -- visualization, distributed databases, simulations, scientific prediction (earthquake), etc.
Silicon based (sequential) architectures reaching physical limits in processing limits as they are constrained by:
the speed of light, thermodynamics

9. Human Architecture! Growth Performance
Vertical
Horizontal
Growth
Age

10. Computational Power Improvement
Multiprocessor
Uniprocessor
C.P.I
No. of Processors

11. Why Parallel Processing?
Hardware improvements like pipelining, superscalar are not scaling well and require sophisticated compiler technology to exploit performance out of them.
Techniques such as vector processing works well for certain kind of problems.

12. Why Parallel Processing?
Significant development in networking technology is paving a way for network-based cost-effective parallel computing.
The parallel processing technology is mature and is being exploited commercially.

13. Parallel Programs
Consist of multiple active “processes” simultaneously solving a given problem.
And the communication and synchronization between them (parallel processes) forms the core of parallel programming efforts.

14. Types of Parallel Systems
Tightly Couple Systems:
Shared Memory Parallel
Smallest extension to existing systems
Program conversion is incremental
Distributed Memory Parallel
Completely new systems
Programs must be reconstructed
Loosely Coupled Systems:
Clusters
Built using commodity systems
Centralised management
Grids
Aggregation of distributed systems
Decentralized management

15. Processing Elements Architecture

16. Processing Elements
Flynn proposed a classification of computer systems based on a number of instruction and data streams that can be processed simultaneously.
They are:
SISD (Single Instruction and Single Data)
Conventional computers
SIMD (Single Instruction and Multiple Data)
Data parallel, vector computing machines
MISD (Multiple Instruction and Single Data)
Systolic arrays
MIMD (Multiple Instruction and Multiple Data)
General purpose machine

17. SISD : A Conventional Computer
Processor
Data Input
Data Output
Instructions
Speed is limited by the rate at which computer can transfer information internally.
Ex: PCs, Workstations

18. The MISD Architecture
Data
Input
Stream
Output
Processor A B C
Instruction
Stream A
Stream B
Instruction Stream C
More of an intellectual exercise than a practical configuration. Few built, but commercially not available

19. SIMD Architecture
Instruction
Stream
Processor A B C
Data Input
stream A
stream B
stream C
Data Output
Ci<= Ai * Bi
Ex: CRAY machine vector processing, Thinking machine cm*
Intel MMX (multimedia support)

20. MIMD Architecture
Instruction
Stream A
Instruction
Stream B
Instruction
Stream C
Data Output
stream A
Data Input
stream A
Processor A
Data Output
stream B
Data Input
stream B
Processor B
Data Output
stream C
Processor C
Data Input
stream C
Unlike SISD, MISD, MIMD computer works asynchronously.
Shared memory (tightly coupled) MIMD
Distributed memory (loosely coupled) MIMD

21. Shared Memory MIMD machine
Processor A
Processor B
Processor C
MEMORY
BUS
MEMORY
BUS
MEMORY
BUS
Global Memory System
Comm: Source PE writes data to GM & destination retrieves it
Easy to build, conventional OSes of SISD can be easily be ported
Limitation : reliability & expandibility. A memory component or any processor failure affects the whole system.
Increase of processors leads to memory contention.
Ex. : Silicon graphics supercomputers....

22. Distributed Memory MIMD
IPC
channel
IPC
channel
Processor A
Processor B
Processor C
MEMORY
BUS
MEMORY
BUS
MEMORY
BUS
Memory
System A
System B
System C
Communication : IPC (Inter-Process Communication) via High Speed Network.
Network can be configured to ... Tree, Mesh, Cube, etc.
Unlike Shared MIMD
easily/ readily expandable
Highly reliable (any CPU failure does not affect the whole system)

23. Laws of caution.....
Speed of computation is proportional to the square root of system cost.
i.e. Speed = Cost
Speedup by a parallel computer increases as the logarithm of the number of processors.
Speedup = log2(no. of processors) C S S P
log2P

24. Caution....
Very fast development in network computing and related area have blurred concept boundaries, causing lot of terminological confusion : concurrent computing, parallel computing, multiprocessing, supercomputing, massively parallel processing, cluster computing, distributed computing, Internet computing, grid computing, etc.
At the user level, even well-defined distinctions such as shared memory and distributed memory are disappearing due to new advances in technology.
Good tools for parallel program development and debugging are yet to emerge.

25. Caution....
There is no strict delimiters for contributors to the area of parallel processing:
computer architecture, operating systems, high-level languages, algorithms, databases, computer networks, …
All have a role to play.

26. Operating Systems for High Performance Computing

27. Types of Parallel Systems
Shared Memory Parallel
Smallest extension to existing systems
Program conversion is incremental
Distributed Memory Parallel
Completely new systems
Programs must be reconstructed
Clusters
Slow communication form of Distributed

28. Operating Systems for PP
MPP systems having thousands of processors requires OS radically different from current ones.
Every CPU needs OS :
to manage its resources
to hide its details
Traditional systems are heavy, complex and not suitable for MPP

29. Operating System Models
Frame work that unifies features, services and tasks performed
Three approaches to building OS....
Monolithic OS
Layered OS
Microkernel based OS
Client server OS
Suitable for MPP systems
Simplicity, flexibility and high performance are crucial for OS.

30. Monolithic Operating System
Application
Programs
System Services
Hardware
User Mode
Kernel Mode
Better application Performance
Difficult to extend
Ex: MS-DOS

31. Memory & I/O Device Mgmt
Layered OS
Application
Programs
Application
Programs
User Mode
Kernel Mode
System Services
Memory & I/O Device Mgmt
Process Schedule
Hardware
Easier to enhance
Each layer of code access lower level interface
Low-application performance
Ex : UNIX

32. Traditional OS OS User Mode Kernel Mode OS Designer Application
Programs
User Mode
Kernel Mode OS
Hardware
OS Designer

33. New trend in OS design Servers Microkernel User Mode Kernel Mode
Application
Programs
Application
Programs
User Mode
Kernel Mode
Microkernel
Hardware

34. Microkernel/Client Server OS (for MPP Systems)
Application
Thread
lib.
File
Server
Network
Server
Display
Server
User
Kernel
Microkernel
Send
Reply
Hardware
Tiny OS kernel providing basic primitive (process, memory, IPC)
Traditional services becomes subsystems
Monolithic Application Perf. Competence
OS = Microkernel + User Subsystems
Ex: Mach, PARAS, Chorus, etc.

35. Few Popular Microkernel Systems
MACH, CMU
PARAS, C-DAC
Chorus
QNX,
(Windows)