Network Systems Lab. Korea Advanced Institute of Science and Technology No.1 Ch. 1 Introduction EE692 Parallel and Distribution Computation | Prof. Song Chong

Network Systems Lab. Korea Advanced Institute of Science and Technology No.2 Issues in Parallelization  Task allocation to processors  The breakdown of the total workload in small tasks assigned to different processors  Proper sequencing of the tasks when some of them are interdependent and cannot be executed simultaneously  Communication of interim computation results between processors  Synchronization of computations of processors  Synchronous: predetermined points for the completion of computations or for the arrival of data  Asynchronous: no such points  Performance measures  Effects of system’s architecture on performance

Network Systems Lab. Korea Advanced Institute of Science and Technology No.3 Need for Parallel & Distributed Computation  Restrict attention to numerical computation in this lecture  Large-scale fast computation  Partial differential eq.s (PDEs) e.g.) fluid dynamics, weather prediction, image processing  Can be decomposed along a spatial dimension  Each processor manipulates the variables associated with a small region in space  Interaction between variables are local in nature  Systems of equations, mathematical programming (optimization)

Network Systems Lab. Korea Advanced Institute of Science and Technology No.4 Need for Parallel & Distributed Computation (Cont’d)  Analysis, simulation and optimization of large scale interconnected systems e.g.) queueing system  Information acquisition, extraction and control in geographically distributed system e.g.) sensor networks, communication networks, wireless networks  Synchronization, unreliable communication, absence of a central control mechanism

Network Systems Lab. Korea Advanced Institute of Science and Technology No.5 Distinction between parallel and distributed computing systems  Parallel computing systems  Processors are located within a small distance of each other  Designed in such a way that processors execute jointly a computational task  Communication between processors is reliable and predictable  Distributed computing systems  Processors may be far apart (geographically distributed)  Communication delay may be unpredictable  Communication links may be unreliable  Topology may undergo changes while operating  Usually loosely coupled, there is very little central coordination and control

Network Systems Lab. Korea Advanced Institute of Science and Technology No.6 Parameters to classify parallel and/or distributed computing systems  Type and number of processors  Presence or absence of a global control mechanism  Synchronous vs. asynchronous operation  Processor interconnections  Tow extremes  Shared memory system solve inter-processor computation problem by lacing a global memory, but introduce the problem of simultaneous accessing of different locations of the memory by several processors  Message-passing system local memory with inter-connection networks