1. Computer Architecture Parallel Processing
Ola Flygt
Växjö University

2. Outline Basic concepts Types and levels of parallelism
Classification of parallel architecture
Basic parallel techniques
Relationships between languages and parallel architecture
CH03

3. Basic concepts The concept of program
ordered set of instructions (programmer’s view)
executable file (operating system’s view)

4. The concept of process OS view, process relates to execution
Process creation
setting up the process description
allocating an address space
loading the program into the allocated address space, and
passing the process description to the scheduler
process states
ready to run
running
wait

5. Process spawning (independent processes)B D E

6. The concept of thread smaller chunks of code (lightweight)
threads are created within and belong to process
for parallel thread processing, scheduling is performed on a per-thread basis
finer-grain, less overhead on switching from thread to thread

7. Single-thread process or multi-thread (dependent)
Thread tree
Process
Threads

8. Three basic methods for creating and terminating threads
1. unsynchronized creation and unsynchronized termination
calling library functions: CREATE_THREAD, START_THREAD
2. unsynchronized creation and synchronized termination
FORK and JOIN
3. synchronized creation and synchronized termination
COBEGIN and COEND

9. Processes and threads in languages
Black box view: T: thread
T2 T T T1 T2 T Tn
FORK
COBEGIN
. . .
FORK
JOIN
COEND
JOIN

10. The concepts of concurrent execution (N-client 1-server)
Non pre-emptive
Pre-emptive
Time-shared
Prioritized
Priority
Sever
Sever
Sever
Client
Client
Client

11. Parallel execution N-client N-server model Synchronous or Asynchronous
Sever

12. Concurrent and Parallel Programming Languages
Classification of programming languages

13. Types and levels of parallelism
Available and utilized parallelism
available: in program or in the problem solutions
utilized: during execution
Types of available parallelism
functional
arises from the logic of a problem solution
data
arises from data structures

14. Available and utilized levels of functional parallelism
Available levels
Utilized levels
User (program) level
User level 2
Process level
Procedure level 1
Loop level
Thread level
Instruction level
Instruction level
1.Exploited by architectures
2.Exploited by means of operating systems

15. Utilization of functional parallelism
Available parallelism can be utilized by
architecture,
instruction-level parallel architectures
compilers
parallel optimizing compiler
operating system
multitasking

16. Concurrent execution models
User level --- Multiprogramming, time sharing
Process level --- Multitasking
Thread level --- Multi-threading

17. Utilization of data parallelism
By using data-parallel architecture
Convert into functional parallelism (i.e. loop constructs)

18. Classification of parallel architectures
Flynn’s classification
SISD
SIMD
MISD (Multiple Instruction Single Date)
MIMD

19. Parallel architectures
Data-parallel architectures
Function-parallel architectures
Instruction-level
PAs
Thread-level
PAs
Process-level
PAs
DPs
ILPS
MIMDs
Pipelined
Superscalar
Vector
Associative
SIMDs
Systolic
VLIWs
Distributed
Shared
and neural
architecture
processors
processors
memory
memory
architecture
architecture
MIMD
(multi-
(multi-computer)
processors)
Part III
Part II
Part IV

20. Basic parallel technique
Pipelining (time)
a number of functional units are employed in sequence to perform a single computation
a number of steps for each computation
Replication (space)
a number of functional units perform multiply computation simultaneously
more processors
more memory
more I/O
more computers

21. Relation between basic techniques and architectures

22. Relationships between languages and parallel architecture
SPMD (Single Procedure Multiple data)
Loop: split into N threads that works on different invocations of the same loop
threads can execute the same code at different speeds
synchronize the parallel threads at the end of the loop
barrier synchronization
use MIMD
Data-parallel languages
DAP Fortran
C = A + B (A, B and C are arrays)
use SIMD
Other types like Vector machines or Systolic arrays does not require any specific language support.

23. Synchronization mechanisms
Test_and_set
Send/receive
message
Net_send
Semaphore
Broadcast
Shift
net_receive
(processor form)
Conditional
Critical region
Monitor
Remote procedure calls
Rendezvous

24. Using Semaphores to handle mutual execution
Busy Wait S
Semaphore
P(S)
P(S)
Critical
region
Critical
region
Shared
data
structure
V(S)
V(S)

25. Parallel distributed computing
Ada
used rendezvous concepts which combines feature of RPC and monitors
PVM (Parallel Virtual Machine)
to support workstation clusters
MPI (Message-Passing Interface)
programming interface for parallel computers

26. Summary of forms of parallelism