1. Institute for Software Science – University of ViennaP.Brezany Parallel and Distributed Systems Peter Brezany Institute for Software Science University of Vienna

2. Institute for Software Science – University of ViennaP.Brezany 2 Typical One-Processor Architecture (SISD Architecture) SISD : Single Instruction stream Single Data stream

3. Institute for Software Science – University of ViennaP.Brezany 3 Array Processor (SIMD Architecture) SIMD: Single Instruction stream Multiple Data streams

4. Institute for Software Science – University of ViennaP.Brezany 4 Loop Parallelizing for SIMDSs A typical scientific program spends approx. 90% of its execution time in loops. Example in Java: float A[1000], B[1000]; for (int i = 1; i < 1000; i++) { A[i-1] = B[i]; } The above loop can be expressed in Fortran 95 in the following way: A(0:998) = B(1:999) This statement can be directly mapped onto a SIMD processor. There is an initiative to extend Java by similar constructs.

5. Institute for Software Science – University of ViennaP.Brezany 5 Parallel Multi-Processor-Hardware (MIMD Architectures) Distributed-memory machines (DM Multiprocessors, DM MIMDS) –Each processor has local memory and disk –Communication via message-passing –Hard to program: explicit data distribution –Goal: minimize communication Shared-memory machines (SM Multiprocessors, SM MIMDs, SMPs) –Shared global address space and disk –Communication via shared memory variables –Ease of programming –Goal: maximize locality, minimize false sharing Current trend: Cluster of SMPs

6. Institute for Software Science – University of ViennaP.Brezany 6 Distributed Memory Architecture (Shared Nothing) Local Memory Local Memory Local Memory Local Memory CPU Interconnection Network CPU

7. Institute for Software Science – University of ViennaP.Brezany 7 DMM: Shared Disk Architecture Local Memory Local Memory Local Memory Local Memory CPU Interconnection Network Global Shared Disk Subsystem

8. Institute for Software Science – University of ViennaP.Brezany 8 Loop Parallelizing for DM MIMDs Example in Java: float A[10], B[10]; for (int i = 1; i < 10; i++) { A[i] = B[i-1]; } For two processors, P1 and P2, a straightforward solution would be:

9. Institute for Software Science – University of ViennaP.Brezany 9 Loop Parallelizing for DM MIMDs (2)

10. Institute for Software Science – University of ViennaP.Brezany 10 Loop Parallelizing for DM MIMDs (3) Code on P1: float A[5], B[5]; float temp; for (int i = 1; i < 6; i++) { if ( i == 5 ) { receive message from P2 into temp; A[i-1] = temp; { else A[i-1] = B[i]; } Code on P2: float A[5], B[5]; float temp; for (int i = 0; i < 5; i++) { if ( i == 0 ) { temp = B[0]; send temp to P1; { else A[i-1] = B[i]; }

11. Institute for Software Science – University of ViennaP.Brezany 11 Shared Memory Architecture (Shared Everything, SMP) CPU Interconnection Network CPU Global Shared Memory

12. Institute for Software Science – University of ViennaP.Brezany 12 Loop Parallelizing for SMPs Example in Java: float A[1000], B[1000]; for (int i = 1; i < 1000; i++) { A[i-1] = B[i]; } If we have, e.g. two processors, P1 and P2, a straightforward (non-optimal) solution would be: Code on P1: for (int i = 1; i < 500; i++) { A[i-1] = B[i]; } Code on P2: for (int i = 500; i < 1000; i++) { A[i-1] = B[i]; } Data elements of A and B are stored in the shared memory.

13. Institute for Software Science – University of ViennaP.Brezany 13 Cluster of SMPs CPU Interconnection Network CPU 4-CPU SMP CPU 4-CPU SMP CPU 4-CPU SMP CPU 4-CPU SMP

14. Institute for Software Science – University of ViennaP.Brezany 14

15. Institute for Software Science – University of ViennaP.Brezany 15 Abstract Maschine Model

16. Institute for Software Science – University of ViennaP.Brezany 16 Cluster von PCs

17. Institute for Software Science – University of ViennaP.Brezany 17

18. Institute for Software Science – University of ViennaP.Brezany 18 Pipeline No Pipeline

19. Institute for Software Science – University of ViennaP.Brezany 19 Towards Parallel Databases

20. Institute for Software Science – University of ViennaP.Brezany 20 Relational Data Model Example Cities NamePopulationLand Munich1211617Bayern Bremen535058Bremen......... Schema: (Cities: STRING, Population: INTEGER, Land: STRING Relation (represented by a table): {(Munich, 1.211.617, Bayern), (Bremen, 535.058, Bremen),...} Key : {Name}

21. Institute for Software Science – University of ViennaP.Brezany 21 Relational Data Model - Queries SELECT FROM [WHERE ] – option.......... other options Example: This is equivalent to: SELECT * SELECT Name, Population, Land FROM Cities SELECT Name, Land FROM Cities WHERE Population > 600000

22. Institute for Software Science – University of ViennaP.Brezany 22

23. Institute for Software Science – University of ViennaP.Brezany 23

24. Institute for Software Science – University of ViennaP.Brezany 24 Grid Idea