Institute for Software Science – University of ViennaP.Brezany Parallel and Distributed Systems Peter Brezany Institute for Software Science University of Vienna

Institute for Software Science – University of ViennaP.Brezany 2 Typical One-Processor Architecture (SISD Architecture) SISD : Single Instruction stream Single Data stream

Institute for Software Science – University of ViennaP.Brezany 3 Array Processor (SIMD Architecture) SIMD: Single Instruction stream Multiple Data streams

Institute for Software Science – University of ViennaP.Brezany 4 Loop Parallelizing for SIMDSs A typical scientific program spends approx. 90% of its execution time in loops. Example in Java: float A[1000], B[1000]; for (int i = 1; i < 1000; i++) { A[i-1] = B[i]; } The above loop can be expressed in Fortran 95 in the following way: A(0:998) = B(1:999) This statement can be directly mapped onto a SIMD processor. There is an initiative to extend Java by similar constructs.

Institute for Software Science – University of ViennaP.Brezany 5 Parallel Multi-Processor-Hardware (MIMD Architectures) Distributed-memory machines (DM Multiprocessors, DM MIMDS) –Each processor has local memory and disk –Communication via message-passing –Hard to program: explicit data distribution –Goal: minimize communication Shared-memory machines (SM Multiprocessors, SM MIMDs, SMPs) –Shared global address space and disk –Communication via shared memory variables –Ease of programming –Goal: maximize locality, minimize false sharing Current trend: Cluster of SMPs

Institute for Software Science – University of ViennaP.Brezany 6 Distributed Memory Architecture (Shared Nothing) Local Memory Local Memory Local Memory Local Memory CPU Interconnection Network CPU

Institute for Software Science – University of ViennaP.Brezany 7 DMM: Shared Disk Architecture Local Memory Local Memory Local Memory Local Memory CPU Interconnection Network Global Shared Disk Subsystem

Institute for Software Science – University of ViennaP.Brezany 8 Loop Parallelizing for DM MIMDs Example in Java: float A[10], B[10]; for (int i = 1; i < 10; i++) { A[i] = B[i-1]; } For two processors, P1 and P2, a straightforward solution would be:

Institute for Software Science – University of ViennaP.Brezany 9 Loop Parallelizing for DM MIMDs (2)

Institute for Software Science – University of ViennaP.Brezany 10 Loop Parallelizing for DM MIMDs (3) Code on P1: float A[5], B[5]; float temp; for (int i = 1; i < 6; i++) { if ( i == 5 ) { receive message from P2 into temp; A[i-1] = temp; { else A[i-1] = B[i]; } Code on P2: float A[5], B[5]; float temp; for (int i = 0; i < 5; i++) { if ( i == 0 ) { temp = B[0]; send temp to P1; { else A[i-1] = B[i]; }

Institute for Software Science – University of ViennaP.Brezany 11 Shared Memory Architecture (Shared Everything, SMP) CPU Interconnection Network CPU Global Shared Memory

Institute for Software Science – University of ViennaP.Brezany 12 Loop Parallelizing for SMPs Example in Java: float A[1000], B[1000]; for (int i = 1; i < 1000; i++) { A[i-1] = B[i]; } If we have, e.g. two processors, P1 and P2, a straightforward (non-optimal) solution would be: Code on P1: for (int i = 1; i < 500; i++) { A[i-1] = B[i]; } Code on P2: for (int i = 500; i < 1000; i++) { A[i-1] = B[i]; } Data elements of A and B are stored in the shared memory.

Institute for Software Science – University of ViennaP.Brezany 13 Cluster of SMPs CPU Interconnection Network CPU 4-CPU SMP CPU 4-CPU SMP CPU 4-CPU SMP CPU 4-CPU SMP

Institute for Software Science – University of ViennaP.Brezany 14

Institute for Software Science – University of ViennaP.Brezany 15 Abstract Maschine Model

Institute for Software Science – University of ViennaP.Brezany 16 Cluster von PCs

Institute for Software Science – University of ViennaP.Brezany 17

Institute for Software Science – University of ViennaP.Brezany 18 Pipeline No Pipeline

Institute for Software Science – University of ViennaP.Brezany 19 Towards Parallel Databases

Institute for Software Science – University of ViennaP.Brezany 20 Relational Data Model Example Cities NamePopulationLand Munich Bayern Bremen535058Bremen Schema: (Cities: STRING, Population: INTEGER, Land: STRING Relation (represented by a table): {(Munich, , Bayern), (Bremen, , Bremen),...} Key : {Name}

Institute for Software Science – University of ViennaP.Brezany 21 Relational Data Model - Queries SELECT FROM [WHERE ] – option other options Example: This is equivalent to: SELECT * SELECT Name, Population, Land FROM Cities SELECT Name, Land FROM Cities WHERE Population >

Institute for Software Science – University of ViennaP.Brezany 22

Institute for Software Science – University of ViennaP.Brezany 23

Institute for Software Science – University of ViennaP.Brezany 24 Grid Idea