1. Introduction to Parallel Processing Chapter No.3

2. Program zProgrammer’s Perspective yan ordered set of instructions zO.S perspective yan executable file stored in secondary memory, typically on a disk.

3. Program

4. Process zFrom OS view, process relates to execution zProcess creation involves 4 major steps: ysetting up the process description yallocating an address space yloading the program into the allocated address space, and ypassing the process description to the scheduler

5. Process Description zO.S describe a process by means of a description table known as process control block (PCB) zPCB contains all the information related to to the whole life cycle of a process like process identification, owner, process status, description of the allocated address space and so on. zDifferent O.S uses different names for process description table like process table in UNIX, Process Information Block in OS/2, and Task control block in IBM mainframe operating systems.

6. Allocating an Address Space zAllocation of an address space to a process can be divided into two steps: ysharing the address among the created processes (shared memory). yAllocating address space to each process (per process address spaces).

7. Loading the Program Into Allocated Address Space zThe executable program file is loaded into the allocated address space and this is done using the different memory management schemes adopted by the O.S.

8. Passing the Process Description to the Scheduler zThe created process is passed to scheduler which allocates the processor to the competing processes. zThis is managed by setting up and manipulating queues of PCBs.

9. The concept of process

10. Process Creation Sub-models zSimpler model used by IBM/PCP, IBM/360/DOS did not allow process to further breakdown into sub-processes. zProcess Spawning ya process may create a new process called a child process. yChild process can be created by the programmer by using standard process creation mechanisms. ySpawned processes form a process hierarchy known as process tree.

11. Process Spawning (Independent Processes) A BC DE

12. Process Scheduling Sub-models zProcess Model yScheduling is performed on per process basis, i.e. the smallest work to be schedules is a process. zProcess-thread Model yin this model smaller units of work is known as thread and they are scheduled as entities.

13. Thread zThread, like a process is a sequence of instructions. zsmaller chunks of code (lightweight) zthreads are created within and belong to process and share the resource of the process. zfor parallel thread processing, scheduling is performed on a per-thread basis zfiner-grain, less overhead on switching from thread to thread.

14. Thread zMost of the O.S released in the 1980’s and 1990’s are based on process thread model such as OS/2, Windows NT or SunOS 5.0. zThreads have similar life cycle to the processes and are mainly managed in the same way.

15. Single-thread Process or Multi-thread (Dependent) Process Threads

16. The Concepts of Concurrent Execution (N-client 1-server) t Simultaneous nature Server Clients Server t Sequential nature Concurrent execution is the temporal behavior of N-client 1 server model where one client is served at any given moment.

17. The concepts of concurrent execution (N-client 1-server) Non pre-emptive Pre-emptive Time-shared Priotized Priority Client Server t Client Preemption rule Non preemptive Time shared preemptive Prioritized

18. Parallel Execution zN-client N-server model zSynchronous: each server starts zservice at the same moment. z Asynchronous: the servers do not work in concert. Servers Clients Servers Synchronous (lock step) Asynchronous

19. Concurrent and Parallel Programming Languages zSequential languages ylanguages that do not support N-client model (C, Pascal, Fortran, PROLOG, LISP) zConcurrent languages yemploy constructs to implement the N-client 1 server model by specifying concurrent threads and processes but lack language constructs to describe N- server model (Ada, Concurrent Pascal, Modula-2, concurrent PROLOG).

20. Concurrent and Parallel Programming Languages zData parallel language yintroduces special data structures that are processed in parallel, element by element (high performance Fortran, DAP Fortran, DAP PROLOG,Connection Machine LISP) zParallel languages yextends the specifications of N-client model of concurrent languages with processor allocation language constructs that enable use of N-server model (Occam-2, 3L Parallel C, Strand-88)

21. Types and levels of parallelism zAvailable and utilized parallelism yavailable: in program or in the problem solutions yutilized: during execution zTypes of available parallelism yfunctional xarises from the logic of a problem solution ydata xarises from data structures that allow parallel operations on their elements, such as vectors or metrices in their problem solution. xGive rise to a parallel execution for data parallel part of computation.

22. Levels of Available Functional Parallelism zParallelism at the instruction level (fine-grained) yparticular instructions of a program executed in parallel. zParallelism at the loop level (middle-grained) yconsecutive loop iterations are candidates for parallel execution. yMay be restricted due to data dependencies between subsequent lop iterations called recurrences. zParallelism at the procedure level (middle-grained) yparallel execution of procedure. zParallelism at the program level (coarse-grained) ymultiple independent users are performing executions in parallel.

23. Available and Utilized Levels of Functional Parallelism Available levelsUtilized levels User (program) level User level Procedure level Process level Loop level Thread level Instruction level Instruction level 1. Exploited by architecture 2. Exploited by means of operating systems 1 2

24. Utilization of Functional Parallelism zAvailable parallelism can be utilized by yarchitecture, xinstruction-level functional parallel architectures or instruction level parallel architectures ( ILP-architectures). ycompilers xparallel optimizing compiler yoperating system xmultitasking

25. Concurrent Execution Models zMultithreading yconcurrent execution at the thread level. yMultiple threads are generated for each process, and these threads are executed concurrently on a single processor under the control of one O.S. zMultitasking yprocess level concurrent execution. zMultiprogramming yuser level concurrent execution. zTimesharing yuser level concurrent execution.

26. Concurrent Execution Models User level Process level Thread level Multiprogramming time-sharing Multitasking Multi-threading

27. Utilization of Data Parallelism zUsing data-parallel architecture ypermits parallel or pipelined operations on data elements.

28. Classification of Parallel Architectures zFlynn’s classification ySISD ySIMD yMISD (Multiple Instruction Single Date) yMIMD

29. Basic Parallel Technique zPipelining (time) yA number of functional units are employed in sequence to perform a single computation. yEach functional unit represent a certain stage of computation. yPipeline allows overlapped execution of instructions. yIt increases the overall processor’s throughput.

30. Car Wash Example Soap cycleDry cycleRinse cycleWax cycle IN Chevy gets soap Rinse station idle Wax station idle Dry station idle Chevy IN Soap station idle Chevy gets rinse Wax station idle Dry station idle Soap station idle Rinse station idle Chevy gets Wax Dry station idle Soap station idle Rinse station idle Wax station idle Chevy gets dry Ford gets soap Rinse station idle Wax station idle Dry station idle Ford IN Chevy OUT

31. Car Wash Example Soap cycleDry cycleRinse cycleWax cycle IN Chevy gets soap Rinse station idle Wax station idle Dry station idle Chevy IN Ford gets soap Chevy gets rinse Wax station idle Dry station idle Ford IN Volvo IN Volvo gets soap Ford gets rinse Chevy gets Wax Dry station idle Saturn IN Saturn gets soap Volvo gets rinse Ford gets Wax Chevy gets dry Toyota gets soap Saturn gets rinse Volvo gets wax Ford gets dry Toyota IN Chevy OUT

32. Replication zReplication (space) ya number of functional units perform multiply computation simultaneously xmore processors xmore memory xmore I/O ymore computers ye.g. array processors.