1. Computer Science and Engineering Parallel and Distributed Processing CSE 8380 April 28, 2005 Session 29

2. Computer Science and Engineering Course Outline 1. Introduction and Motivation 2. Architecture and Performance 3. Algorithms and Programming 4. Task Scheduling 5. Advanced Topics

3. Computer Science and Engineering Parallel and Distributed Processing Uniprocessor Multiprocessor Speed-upQuality-upSharing Physical limitations N processors cooperate to solve a single computational task

4. Computer Science and Engineering Motivation Uniprocessor systems are not capable of delivering solutions to some problems in reasonable time Multiple processors cooperate to jointly execute a single computational task in order to speed up its execution Speed-up versus Quality-up

5. Computer Science and Engineering Parallel versus Sequential Computing Multiple threads of control vs. single thread of control Partitioning for concurrent execution Task Scheduling Synchronization Performance

6. Computer Science and Engineering A model is an interface separating high level properties from low level ones Modeling Applications Architectures Provides operations Requires implementation MODEL

7. Computer Science and Engineering Leopold’s View of the Field Numerous Application Programs Concrete Architectures PthreadsJava Threads OpenMP Skeletons MPI PVM Threads Shared Memory Message Passing Distributed SM Cluster SMPCC-NUMAATMMyrinet Hiding Details High Low

8. Computer Science and Engineering 3 Layers Applications Parallel Tools Architecture Smart Compiler

9. Computer Science and Engineering Course Outline 1. Introduction and Motivation 2. Architecture and Performance 3. Algorithms and Programming 4. Task Scheduling 5. Advanced Topics

10. Computer Science and Engineering Past Trends in Parallel Architecture (inside the box) Completely custom designed components (processors, memory, interconnects, I/O) Longer R&D time (2-3 years) Expensive systems Quickly becoming outdated Bankrupt companies!!

11. Computer Science and Engineering New Trends in Parallel Architecture (outside the box) Advances in commodity processors and network technology Network of PCs and workstations connected via LAN or WAN forms a Parallel System Network Computing Compete favorably (cost/performance) Utilize unused cycles of systems sitting idle

12. Computer Science and Engineering Architecture Three major Components Processors Memory Modules Interconnection Network

13. Computer Science and Engineering Parallel and Distributed Computers MIMD Shared Memory Bus based Switch based CC-NUMA MIMD Distributed Memory SIMD Computers Clusters Grid Computing

14. Computer Science and Engineering MIMD Shared Memory Systems Interconnection Networks MMMM PPPPP

15. Computer Science and Engineering MIMD Distributed Memory Systems Interconnection Networks MMMM PPPP

16. Computer Science and Engineering SIMD Computers Processor Memory P M P M P M P M P M P M P M P M P M P M P M P M P M P M P M P M von Neumann Computer Some Interconnection Network

17. Computer Science and Engineering Clusters M C P I/O OS M C P I/O OS M C P I/O OS Middleware Programming Environment Interconnection Network

18. Computer Science and Engineering Grids Grids are geographically distributed platforms for computation. They provide dependable, consistent, pervasive, and inexpensive access to high end computational capabilities.

19. Computer Science and Engineering Interconnection Network Taxonomy Interconnection Network Static Dynamic Bus-basedSwitch-based 1-D2-DHC SingleMultiple SSMS Crossbar

20. Computer Science and Engineering Course Outline 1. Introduction and Motivation 2. Architecture and Performance 3. Algorithms and Programming 4. Task Scheduling 5. Advanced Topics

21. Computer Science and Engineering Speedup, Efficiency, Utilization Amdahl’s Law The Gustafson-Barsis Law Benchmarks Performance Evaluation

22. Computer Science and Engineering Course Outline 1. Introduction and Motivation 2. Architecture and Performance 3. Algorithms and Programming 4. Task Scheduling 5. Advanced Topics

23. Computer Science and Engineering PRAM Model Synchronized Read Compute Write Cycle EREW ERCW CREW CRCW Complexity: T(n), P(n), C(n) Control Private Memory P1P1 Private Memory P2P2 Private Memory PpPp Global Memory

24. Computer Science and Engineering Parallel Algorithms Sorting on CRCW PRAM Computing sum on EREW PRAM Computing all partial sums on EREW PRAM Matrix Multiplication on CREW Other Algorithms

25. Computer Science and Engineering Distributed Algorithms Message Passing Model Complexity Analysis Leader Election

26. Computer Science and Engineering Course Outline 1. Introduction and Motivation 2. Architecture and Performance 3. Algorithms and Programming 4. Task Scheduling 5. Advanced Topics

27. Computer Science and Engineering Parallel Virtual Machine (PVM) Environment & Application Structure Task Creation Task Groups Communication Synchronization Reduction operations Work Assignments

28. Computer Science and Engineering Course Outline 1. Introduction and Motivation 2. Architecture and Performance 3. Algorithms and Programming 4. Task Scheduling 5. Advanced Topics

29. Computer Science and Engineering Task Scheduling Model Program tasks Machine Schedule Execution and communication time Problem Complexity

30. Computer Science and Engineering Scheduling System ConsumersResourcesScheduler Policy

31. Computer Science and Engineering Task Graph A 10 D 15 E 10 F 20 B 15 C 10 G 15 H I 30 5 5 87 5 55 10 5 4 54 20

32. Computer Science and Engineering Scheduling Algorithms Optimal Algorithms (w/o communication) 1. Task graph is in-forest or out-forest 2. Task graph is an interval order 3. two processors Optimal Algorithms (with communication) 1. Task graph is in-forest or out-forest on 2 procs 2. Task graph is an interval order Many heuristics

33. Computer Science and Engineering Course Outline 1. Introduction and Motivation 2. Architecture and Performance 3. Algorithms and Programming 4. Task Scheduling 5. Advanced Topics

34. Computer Science and Engineering Good Luck to You!!